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ABSTRACT 


There has been much concern in recent years about the rapid increase in 
automation on commercial flight decks. Part of the concern is that automation is 
introduced without an explicit guiding philosophy about its usability and its 
impact on overall flight crew-flight deck performance and aircraft safety. Several 
surveys of commercial airline pilots have addressed flight deck automation 
issues. However, most have solicited opinions specific to automation 
implemented on particular aircraft. This paper describes a survey that was 
aimed at gathering pilot opinions about high level, automation philosophy 
issues. It was administered to 132 pilots of advanced automation aircraft. The 
respondents included 46 Airbus A-320 pilots (representing Northwest, United, 
and America West), 47 Boeing 747-400 pilots (representing Northwest and 
United), and 39 Douglas MD-11 pilots (representing American and Delta). 

The survey was composed of four major sections. The first section asked pilots to 
rate different automation components that exist on the latest commercial aircraft 
regarding their obtrusiveness and the attention and effort required to use them. 
The second section addressed general "automation philosophy" issues, such as: 
What attributes make automation "good?" What attributes make it 
"trustworthy?" Has increased automation increased or decreased physical and 
mental workload? The third section focused on issues related to levels and 
amount of automation, such as: On the continuum from unaided pilot 
performance to completely automated functions, where should flight deck 
systems lie — in different situations and for different functions? What are the 
advantages and disadvantages of different levels of automation? The fourth 
section addressed several design issues specific to a next-generation high speed 
dvil transport aircraft. 

The results indicate that pilots of advanced aircraft like their automation, use it, 
and would welcome more. However, they also believe that automation has 
disadvantages, especially fully autonomous automation. They want their 
automation to be simple and reliable and to produce predictable results. 
Furthermore, in response to questions about the extent to which they felt in 
control of the aircraft versus controlling the automation itself, they revealed that 
simple and reliable is, in some ways, related to how little attention they need to 
pay to it. If there is a necessity to interact with automation extensively, such as 
with systems requiring intensive data input or components that must be 
constantly monitored, it is more likely to be perceived as obtrusive and pilots' 
attention will be focused on the automation instead of the underlying function. 
This phenomenon may be an unavoidable penalty for human-centered 
automation when contrasted with autonomous systems and highlights the 
importance of effective interface design. 
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Although we began this survey with the objective of trying to understand the 
contrast between human-centered and full automation, we come away from it 
with a slightly different perspective. To be against human-centered automation 
is to be against apple pie. The issues instead are to understand, from the pilots 
point of view, how far they want to go in introducing automation and what 
features need to be present to maintain situation awareness, to assure human 
control of the integrity of flight and to promote safety and airline cost- 
effectiveness. The answer provided in this survey is that the greatest promise for 
further gains with the fewest disadvantages is obtained in moving from manual 
systems to shared-performance systems, as contrasted with moving from shared- 
performance systems to autonomous systems. 

Although pilots generally indicated they would like more of all types of 
automation, the biggest needs for higher levels of automation were in pre-flight; 
communication, systems management, and task management functions* 
planning as well as response tasks; and high workload situations. There is an 
irony and a challenge in the implications of these findings. On the one hand 
pilots would like new automation to be simple and reliable, but they need it to 
support the most complex part of the job— managing and planning tasks, 
especially in high workload situations. 
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INTRODUCTION 


There has been much concern in recent years about the rapid increase in 
automation on commercial flight decks. The results of pilot surveys have raised 
flags over such issues as increased heads down time, degradation of flying skills, 
workload extremes, and the requirement to manage unanticipated situations' 
(e.g., McClumpha, James, Green, & Belyavin, 1991; Sarter & Woods, 1991; 
Wiener, 1989). The factors underlying these findings have been explored by 
researchers and industry experts (e.g., ATA, 1989; Kantowitz, & Sorkin, 1987; 
Norman, 1986; Regal & Braune, 1992; Wiener & Curry, 1980). Part of the concern 
is that automation has been introduced without an explicit guiding philosophy 
about its usability and its impact on overall flight crew-flight deck performance 
and aircraft safety. "The fundamental concern," according to the Air Transport 
Association of America, "is the lack of a scientifically-based philosophy of 
automation that describes the circumstances under which tasks are appropriately 
allocated to the machine and/or the pilot" (ATA, 1989). To address this need, the 
ATA has created a "National Plan to Enhance Aviation Safety through Human 
Factors Improvements. In a similar vein, NASA has initiated an Aviation 
Safety/ Automation Program that has made the development of a flight deck 
automation philosophy a high priority (Billings, 1991; Norman & Orlady, 1989). 

The challenge in developing an automation philosophy has been well described 
by Riley (1989): 

In the past, automation decisions could be made by assigning to human or machine 
whatever task each was better able to perform. Now, this criterion is often difficult to 
apply. Furthermore, designers must consider such issues as the operator’s loss of 
situation awareness due to relying too heavily on automation, conflicts between the 
operator's decisions and the machine’s decisions, or tendencies of the operator to 
override or defeat the automation due to lack of trust in it, p.124. 

Of particular relevance to these issues is the growing literature on the cognitive 
processes involved in the management of complex multi-task systems (see 
Adams, Tenney, and Pew, 1994, for a review). With it, has come an appreciation 
of the difficulties faced by the operator: 


The distinctive characteristic of the complex semi-automated environment is that the 
operator is confronted with a number of tasks at once. No matter: the operator's 
behavior must still be goal directed. In order to switch in and out of tasks so as to 
maintain the integrity of the system or mission as a whole, the operator must 
subordinate each of the individual tasks and the overall ensemble to his or her 
understanding of the goals of the system or mission as a whole. Given that the 
opportunity, urgency, and particulars of completing any given task must be in 
constant flux with the stream of events, this is no small requirement. (Adams 
Tenney, & Pew, 1994, p. 35). 

One implication of this research is that automation should be "human-centered" 
(Billings, 1991; Rouse, Geddes, & Curry, 1987; Wiener, 1989). At a minimum, it 



must provide a coherent task breakdown, maintain focus on higher-level goals, 
minimize distractions, and offer strong contextual support following 
interruptions (Adams, Tenney, & Pew, 1994). 

Another way to interpret this literature, however, is that humans, with their 
propensities for error and cognitive limitations, should be taken out of the loop 
altogether. Why trust any function to a human when a machine can be 
programmed to do it better? These two views, the human-centered and its 
opposite, the fuU-automation view, are two candidate design philosophies that 
have been discussed in the human factors literature (Billings, 1991; Norman & 
Qrlady, 1989; Regal & Braune, 1992; Rouse, Geddes, & Curry, 1987). 

To date, there has been little support among researchers for the full-automation 
philosophy, for both technical and political reasons (Regal & Braune, 1992). The 
human centered-view, by contrast, has many proponents (e.g., Billings, 199 , 
Norman & Orlady, 1989; Regal & Braune, 1992; Wilson & Fadden, 1991). It is 
founded on the premise that "the flight crew will be an integral component of 
safe and efficient commercial flight for the foreseeable future because human 
skills, knowledge, and flexibility are required in the operation of complex 
systems in an unpredictable and dynamic environment" (Palmer, Rogers, Press, 
Latorella, & Abbott, 1995). As a consequence, the crew must be kept involved in 
and informed about all aspects of the flight situation that they would need to 
know about if they were to fly manually and must always have the option to do 

so. 

Billings (1991) has described the principles of human-centered automation as 
follows The basic premise is: "The pilot bears the ultimate responsibility for the 
safety of any flight operation" (Billings, 1991). An axiom of this premise is: "The 
human operator must be in command." Corollaries to this axiom include. 

To command effectively, the human operator must be involved. To be involved, the 
human operator must be informed. The human operator must be able to monitor the 
automated systems. Automated systems must therefore be predictable. The 
automated systems must be able to monitor the human operator. Each element of the 
system must have knowledge of the others' intent (Billings, 1991, p. 12). 

The goal, in this view, is to create automation that supports human strengths and 
compensates for human weaknesses, while leaving ultimate control in the hands 
of the human (Billings, 1991; Regal & Braune, 1992). The concept of providing 
different levels of automation for flight control— from highly automated control 
through a flight management computer, to moderately automated control 
through a mode control panel, to basically manual control through a wheel and 
column and throttle— was an initial step in this direction. Today, there are many 
systems, either on the flight deck or on the drawing board, that would qualify as 
human-centered, in the sense of easing information processing, enhancing 
situation awareness, and preventing mistakes. Examples include graphic map 
displays, integrated caution and warning systems, decision aids, electronic 
checklists, and electronic libraries. 
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Although the human-centered approach may sound good in theory, in practice 
there are many unresolved questions. How much knowledge should a pilot be 
expected to have? Are there times and situations where advising the human and 
waiting for a decision would be riskier than having the machine take direct 
action? Is it conceivable that in the future there will be aspects of flight that are 
so well understood that they can be programmed to be completely reliable and 
risk-free without the need for human monitoring? These issues are not just 
limited to flight decks. They are issues whenever automation is seen as 
"simplifying" large complex systems. They point, once again, to the need for 
careful guidelines: "A well thought-out philosophy will help designers achieve 
the most effective balance between fully automated and human-centered 
automation systems" (Regal & Braune, 1992). 

The purpose of this study was to contribute to the development of a design 
philosophy for new, advanced flight decks by gathering data on pilots' views on 
automation philosophy as well as on some of the many concepts and definitions 
that will be needed to formulate meaningful guidelines: for example, what it 
means for automation to be trustworthy, how to define levels and types of 
automation, and how to characterize good and bad automation experiences. This 
information was collected by administering a survey concerned with high-level 
automation issues to commercial airline pilots. 

The Survey 

The present survey was administered to captains and first-officers of the Boeing 
747-400, Douglas MD-11, and Airbus A-320 aircraft. These aircraft were selected 
because, when the survey was administered, they were the newest flight decks of 
each manufacturer that had been in service long enough to have a sufficient 
number of experienced respondents. The Airbus A-320 went into revenue 
service in 1988, the Boeing B747-400 in 1989, and the Douglas MD-11 in 1991. An 
advantage of using all three was that they provided the opportunity to compare 
responses along a technological dimension. The Airbus A-320 is considered to be 
the most highly automated. For example, it is a fly-by-wire aircraft and has 
envelope protection features that cannot be overridden. Boeing, is perceived to 
be the least automated of the three. For example, its envelope protection system 
advises, but does not restrict, the pilot. Douglas, in turn, is considered to lie 
somewhere in between the other two. (However, systems management on the 
MD-11 is the most automated of all three aircraft.) By including pilots of all three 
flight decks in the survey it was possible to look for differences in attitudes and 
opinions as a function of different automation implementations on the aircraft 
they fly. 

Philosophy 

The first issue addressed in the survey was the extent to which pilots endorsed 
the human-centered, as opposed to the fully automated philosophy. While 
researchers may be leaning strongly toward a human-centered philosophy, it 
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was not clear to what extent the actual users and designers of the automation 
shared their views. Regal & Braune (1992) have warned: 

It appears that, at the present time, many individuals and groups find it easier to work 
on developing fully automated systems than dealing with the difficulties involved in 
tailoring systems to the cognitive complexities of humans. We need to understand the 
advantages and disadvantages of a human-centered automation approach so that it is 
possible to objectively choose the optimal system design, p. 3. 

A human-centeredness scale, consisting of thirteen questions, was devised by 
combing the literature for statements that reflected either a human-centered or 
full-automation view and then pairing each statement with an opposing 
statement (see Appendix A, Q32-41 and Q46-48 and Table 1). As in previous 
surveys the paired statements were presented to subjects as the endpoints of a 
scale (McClumpha et al., 1991). Subjects could indicate strong agreement with 
either statement by choosing a scale point close to one of the extremities, mild 
agreement by choosing a point just on either side of the midpoint of the scale, or 
neutrality, by choosing the midpoint itself. Table 1 identifies the endpoints for 
each question as human-centered or fully-automated and references the sources 
of the ideas for the questions. 

Good/Trustworthv Automation 

The second issue on which pilot opinions were sought concerned the attributes 
that are most essential for making pilots like and trust the automation on their 
flight deck. Billings (1991) has proposed a set of desirable attributes for human- 
centered automation. As he pointed out, however, it may not be possible to give 
equal emphasis to all the attributes. For example, an emphasis on 
informativeness, by increasing the complexity of the machine, might be 
incompatible with maximal dependability. For that reason, it seemed 
worthwhile to solicit pilot opinions on those attributes that should receive top 
priority. Therefore, two questions were composed in which subjects were asked 
to rank order the ten attributes proposed by Billings and an eleventh one 
"simple"— that appeared to the investigators to be curiously missing (see 
Appendix A, questions 51 and 52). In question 51, subjects ranked the attributes 
with respect to good automation (defined as enhancing safe, economical operation 
of the aircraft). In question 52 they ranked them with respect to trustworthy 
automation (defined as producing pilot acceptance and faith in its operation or 
output). 
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Table 1 


Human-Centered Philosophy Questions 

Q32-41; 46-48 


Quest- 
ion No. 

Opening 

Human-Centered 

Ending 

Fully- Automated 
Ending 

Source of 
Idea for 
Question 

32 

Initial emphasis in 
training should be 
placed on learning about 

the "basic airplane" before 
introducing the 
automation 

the automation before 
introducing the basic 
airplane 

Norman k Or lady, 
1989, p. 184) 

33 

I would like to see the 
introduction of more 
automation that 

assists the pilot in problem 
solving 

automatically solves 
problems 

Morgan Herschler, 
Wiener, k Salas 
(1995) 

34 

I would like to see the 
introduction of more 
automation that 

evaluates and advises the 
flight crew on alternative 
plans of action 

automatically executes 
alternative plans of actions 

Morgan, Herschler, 
Wiener, k Salas 
(1995) 

35 

It is likely that 
situational information 
about A/C systems 
given by automation in 
the future 

will always require 
confirmation 

can always be accepted as 
fact 

Pew (1988) 

36 

It is likely that 
procedural info, 
concerning A/C systems 
given by automation in 
the future 

will always require 
confirmation 

can always be accepted as 
fact 

Pe w (1988) 

37 

It is likely that the pilot 
in the future will 

still be responsible for 

flying the 

aircraft 

not be responsible for 

flying the 

aircraft 

Billings (1991, p. 12, 
57, 80) Jordan (1963, 
p. 162) 

38 

In the future 

there will still be system 
failures that require 
deviations from standard 
procedures 

we can expect that every 
system failure will have a 
prescribed procedure to 
follow 

Regal k Bra une 
(1992, p.4) 

39 

The biggest obstacle to 
total flight safety is 

system design 

pilot performance 

Pew (1988) 

40 

In pilot training in the 
future, principles of 
navigation and aviation 
are likely to receive 

just as much 
emphasis 

less emphasis 

Norman k Orlady 
(1989) 

41 

In most cases, an 
automatic system should 

warn the crew of envelope 
exceedence, but not 
restrict pilots’ control 

prevent the aircraft from 
exceeding its performance 
envelope 

Billings (1991, p. 29, 
86) 

46 

Decision aids should 
emphasize 

situation 

information 

response 

information 

Morgan, Herschler, 
Wiener, k Salas 
(1995) 

47 

Decision aids should 
provide the flight crew 
with 

a list of alternatives 

one alternative 

Morgan, Herschler, 
Wiener, k Salas 
(1995) 

48 

Decision aids should 
provide the flight crew 
with 

recommendations 

commands 

Morgan, Herschler, 
Wiener, k Salas 
(1995) 
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Physical and Mental Workload 

The third issue concerned the nature of workload on the advanced flight deck. 
Previous surveys have produced somewhat contradictory results. Some studies 
have reported workload problems, particularly workload extremes, while others 
have not. Wiener (1989), found that automation increased workload at the 
busiest times, but this result was not supported by McClumpha et al. (1991). In 
order to explore this issue more fully, subjects were asked to rate both physical 
and mental workload separately for each phase of the flight (see Appendix A, 
question 55). 

Levels of Automation: Promise and Concerns 

The fourth issue related to levels of automation, that is, the degree to which 
functions or tasks are performed by the pilot, the automation, or some 
combination. The question was whether pilots could evaluate different levels of 
automation on the basis of their costs and benefits (see Appendix A, questions 
78-80). While previous surveys have collected pilots' expectations and concerns 
about automation in advanced, in contrast to traditional flight decks, they have 
not, to our knowledge, been asked to consider the pros and cons of different 
levels of automation. 

How many, and what kinds of levels, are meaningful to pilots? Again, the goal 
was to establish the psychological validity of the concept of a level, while at the 
same time collecting data about the trade-offs between costs and benefits 
associated with the different levels. The ability of subjects to assign different 
expectations and/or concerns to the different levels would provide evidence that 
the distinction between levels was real to them. On the other hand, if subjects 
gave statistically indistinguishable responses to the three levels, it would be 
difficult to argue that the levels were meaningful. 

In developing this survey, our original intent was to gather data on all six levels 
of automation proposed by Billings (1991). Billings' automation levels not only 
distinguished whether the flight crew or automation, or some combination, was 
assigned a particular function or task, but also whether the pilot or the 
automation had authority over task performance, and if the automation had 
authority, how that authority was delegated by the pilot. Closer examination of 
these levels, however, revealed that some of the distinctions among levels were 
subtle and ambiguous. We were concerned that opinions about advantages and 
disadvantages of different levels, or what the ideal level was for different 
situations, would be unreliable given the difficulty of understanding the 
differences among levels. 

Furthermore, as Regal & Braune (1992) have observed, different researchers have 
proposed different schemes. The most extensive is from Riley (1989), who has 
proposed seven levels along a "machine intelligence" dimension (ranging from 
"raw data" to "the anticipation of operator error") and twelve levels along an 
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"autonomous" dimension (ranging from varying degrees of communication with 
the pilot to varying degrees of taking action with or without pilot permission or 
overrides). Thus it is evident that many dimensions (e.g., task performance, 
authority, machine intelligence) underlie different notions of automation levels. 

Rather than attempting to examine the whole problem of levels, which we 
suspect, along with Riley (1989), will be multidimensional and complex, we 
opted, as a first step, to see if a meaningful distinction could be made between 
one intermediate level of automation, "shared pilot/ automation performance," 
and the two extremes: "fully autonomous" and "unassisted pilot performance" 
(see questions 78-80 for definitions). In these questions, subjects were presented 
with a list of specific concerns and expectations that had appeared in the 
literature (ATA, 1989; McClumpha et al., 1991; Norman & Orlady, 1989, 
Appendix A; Parasuraman, Molloy, & Singh, 1993; Pew, 1988; Singh, Molloy, & 
Parasuraman, 1993; Wiener, 1989) and were asked to indicate how strongly they 
felt each one applied to each of the three levels of automation. 

Ideal Level of Automation for Different Situations 

The fifth issue addressed in the survey was pilots' satisfaction with the level of 
automation available on their aircraft for particular situations (See Appendix A, 
questions 56-77). It is common to hear concerns that aircraft are too automated 
or that automation has too much authority, yet pilots applaud many of the most 
sophisticated automated systems on their aircraft and express the need for more 
automated assistance in some situations. We argue that the ideal level of 
automation cannot be judged independently of the particular situation or 
context. For example, anecdotal reports from pilots suggest that they feel under- 
utilized during normal conditions, due to the amount of automation, but could 
use more automation in non-normal conditions. The question then becomes, 
what aspects or attributes of the situation affect the ideal level of automation 
from the pilot's perspective? Recent work on defining situation awareness 
requirements (Deutsch, Pew, Rogers, & Tenney, 1994) suggests that it is useful to 
decompose complex situations into simple situations that vie for the pilot's 
attention and to decompose those situations into a set of attributes. In the 
context of commercial aviation, the attributes that appear to be most important 
for defining situations are the environmental and system events that occur and 
the tasks and functions that can be carried out. In questions 56 to 77, subjects 
were asked to consider the following situation attributes: flight condition (i.e., 
normal, non-normal, emergency); phase of flight; mission function (i.e., flight 
control, navigation, communication, systems management, and task 
management); human information processing task (i.e., monitoring, processing, 
responding); and workload (i.e., high, medium, low) (Deutsch et al., 1994; Regal 
& Braune, 1992). 

For each category of each situation attribute (e.g., for "high workload" in the 
workload attribute; for "navigation" in the mission function attribute), subjects 
were asked to indicate: a) the level of automation they had available on their 
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aircraft for that item; and b) the level they would use ideally. They made each of 
these judgments on a five point scale, ranging from "unassisted pilot 
performance" to "totally automated performance." By comparing actual vs. ideal 
level it was possible to see in what situations differences between current and 
ideal levels of automation exist. Equally importantly, subjects’ ability to 
differentiate between the different categories of a particular situation attribute 
would suggest that the particular ways of decomposing situations, and the scales 
used, were meaningful. It is important to establish the psychological validity of 
these categories because the formulation of automation guidelines is likely to 
involve this intermediate level of discourse (e.g.. For the cruise phase of flight, or 
for low workload situations, the ideal level would be shared performance). 

Amount of Automation 

The sixth issue again concerned the question of whether the flight deck is over- 
or under-automated. Because this issue is such an important one it seemed 
worthwhile to approach it in different ways. This time instead of estimating 
ideal levels of automation for different situations, subjects were asked to rate the 
ideal amount of automation, where amount was defined as the total number of 
automated systems or components on the aircraft. 

Subjects answered five questions, concerning five different types of automation. 
Each question had two parts. In part 'a' they rated the amount of each type of 
automation in their current aircraft. In part V they indicated whether the ideal 
amount of automation would be more than, less than, or the same as in their 
current aircraft . 

The questions differed from those in the previous section in the following ways: 

1 . The questions concerned amount rather than level of automation. 

2. The questions asked about a different set of categories. This time subjects 
were asked to consider five types of automation: aircraft control, systems 
control, information automation, decision automation, and protective 
automation (see Q81-85 for definitions). These categories were derived from 
Billings' (1991) distinction between control, information, and management 
automation and were expected to have psychological validity. 

3. In part V subjects indicated how far they thought they were from the ideal 
am ount of automation. This time there was no need to subtract the responses 
of part / b' from 'part a' because subjects compared the actual to the ideal 
directly in / b.' (Another reason for not subtracting was that the two scales 
were not equivalent.) 

Phenomenological Experiences 

The sixth and final automation issue concerned the question of how to talk about 
the phenomenological experience of using automation. Dimensions of 
experience that have been emphasized in the literature and are clearly of 


8 



relevance to designers are the notion of workload (e.g., Adams, Tenney, & Pew, 
1994; Gopher & Donchin, 1986; Wickens, 1992) and situation awareness (e.g., 
Adams, Tenney, & Pew, 1995; Endsley, 1995; Gilson, Garland, & Koonce, 1994; 
Sarter & Woods, 1991; Taylor, 1989; Taylor & Selcon, 1993). There is another 
dimension of experience, however, that has been noted persistently in the 
literature, but whose ramifications have not yet been explored (Billings, 1991; 
Hollnagel, 1991). It has to do with the feeling that one is having a direct as 
opposed to an indirect, or mediated experience. An example of a direct 
experience would be the feeling, when driving, that one can sense the bumps on 
a road directly, rather than through the tires. The same distinction is believed to 
occur in aviation. Pilots may feel that they are controlling certain aspects of the 
flight directly, even though they are assisted by automation. On the other hand, 
they may become so involved with the demands of the automation, that their 
attention is completely diverted from the flight. In such cases, pilots may feel 
that they are managing or supervising the automation rather than controlling the 
flight. This kind of indirect experience is believed to occur when the automation 
is new, cognitively demanding, or otherwise obtrusive (Hollnagel, 1991). 

In order to explore this dimension, subjects were presented with a list of 31 flight 
deck automation systems they were likely to have encountered and were asked 
to rate each one on several different dimensions of experience, including 
situation awareness, or "knowledge of the big picture,” workload, the 
controlling/ managing dimension, distractibility, predictability, and frequency of 
use (see Appendix A, questions 1-31). (Slightly different scales were used for the 
alerting questions, 27-31). 

The list was compiled based on components described by Billings (1991) and 
McGuire et al. (1991). It was then reviewed by two retired airline pilots. They 
added and subtracted items based on whether the items made sense in terms of 
the scales subjects would use to rate them and whether they were described in 
language that would be unambiguous to pilots. Some items, mostly ones that 
were either completely transparent or required no pilot interaction, were 
eliminated based on preliminary data indicating that they would produce 
uniform responses. 

Results were analyzed to see if the dimensions clustered into sub-dimensions 
and to see which pieces of automation gave rise to which types of experience. 
The results of the questions relating to phenomenological experiences will be 
described in the results section, but the supporting tables summarizing the data 
are provided in Appendix B. 

Miscellaneous 

Included in the questionnaire for other purposes were some questions designed 
to assess preferences for specific design options. Two questions (see Appendix 
A, questions 42,43) related to how autopilot modes are organized. The 
functioning of the autoflight systems differs among aircraft types. In terms of a 
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philosophy, it has been suggested that the autoflight modes should reflect what it 
is that pilots really want to control, that is, speed, lateral path and vertical path. 
Conventional systems typically are organized by thrust, roll, and pitch, the 
aircraft parameters that are directly affected by control inputs. Secondly, given 
that autoflight modes have become very complex and are suspected as factors in 
several recent accidents, a question about the predictability of mode transitions 
was asked. A third question addressed the use of synthetic speech (see 
Appendix A, question 44). Synthetic speech is seen as a technology that can 
facilitate presentation of information to the flight crew, particularly in high visual 
workload conditions, but commercial flight decks have traditionally been very 
conservative in regard to introduction of such technologies. A fourth question 
related to the overall amount of information available on the flight deck (see 
Appendix A, questions 45); the question has continually been raised as to 
whether there is too much information on today's flight decks, but to our 
knowledge a pilot sample has never been directly asked. Two questions address 
decision aids and the presentation of probabilistic information (Q49,50). As 
decision aids and artificial intelligence technologies continue to advance, and 
automation becomes more capable of probabilistic assessments, the question 
arises as to how certain information should be before it is presented to the pilot. 
Finally, seven questions (Q88-94) addressed issues related to the ongoing design 
of the High Speed Civil Transport (Alter & Regal, 1991; Regal & Alter, 1993; 
Swink & Goins, 1992). These included questions concerning the use of displayed 
sensor and database data to replace forward vision lost due to the elimination of 
forward windows, and questions concerning the best control device, that is, a 
side stick, wheel and column, or center stick. The results of these questions will 
be presented and discussed in Appendix C. 1 

METHOD 


Subjects 

Captains and First Officers actively flying Boeing B747-400, McDonnell Douglas 
MD-11, and Airbus A-320 aircraft for U.S. airlines were recruited for this survey. 
A total of 132 pilots, 47 B747-400 pilots, 39 MD-11 pilots, and 46 A-320 pilots, 
completed the survey. The sample included one female. The distribution of 
Captains and First Officers across airlines and aircraft types is shown in Table 2. 


1 Also included, for exploratory purposes, were four open-ended questions (Q 53, 54, 86, 87). 
The results will not be discussed in this report. 
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Table 2 

Number of Captains and First Officers Completing the Survey by Aircraft 

Type and Airline 


Aircraft Type 

Airline 

Seat 

Number of Pilots 

B747-400 

Northwest 

Captains 

2 

(N=47) 


First Officers 

17 


United 

Captains 

3 



First Officers 

25 

MD-11 

American 

Captains 

11 

(N=39) 


First Officers 

2 


Delta 

Captains 

13 



First Officers 

13 

A-320 

Northwest 

Captains 

5 

(N=46) 


First Officers 

2 


United 

Captains 

10 



First Officers 

11 


America West 

Captains 

8 



First Officers 

10 


The mean age of pilot subjects was 45.9 with a range of 28 to 59. Subjects 
averaged 18.9 years of commercial flying experience, 12704 total flying hours, 
and 15.8 years of formal education. All subjects had experience on jet aircraft 
other than the one for which they were currently type rated. Classification of 
three of the subjects, in terms of aircraft type, was problematic because they were 
currently type-rated on both the Airbus A-320 and the Boeing 747-400. In those 
cases, subjects were assigned to the aircraft in which they indicated more flying 
hours. 2 

It is apparent from Table 2 that for unknown, assumably random reasons, there 
was an unequal distribution between Captains and First Officers. The proportion 
of First Officers in the groups varied as follows: Boeing 89%, Douglas 38%, and 
Airbus 50%. Table 3 shows the background characteristics of subjects in the three 
aircraft groups. These data were compiled from the biographical forms that each 
subject filled out. Unfortunately, due to the unequal representation of Captains 
and First Officers across aircraft types, differences emerged in their background 


2 One subject in this category was inadvertently misclassified as Boeing instead of Airbus. 
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characteristics, complicating the task of comparing the aircraft groups . As can 
be seen in the table, Douglas pilots were, on the average nine or ten years older 
than the other pilots and, as a result, had more total flying hours and more hours 
flying as pilot in command. The analyses of variance for these characteristics 
were all significant: F(2,126)=23.23, p<.0001 for age, F(2,127)=14.93, pc.OOOl for 
total flying hours, and F(2,125)=10.94, pc.OOOl for hours as pilot in command. 
Tukey tests revealed that in all cases, the Douglas pilots differed from the other 
two groups (p<.05), which did not differ from each other (p>.05). 

Table 3 

Biographical Data for Pilots of Different Aircraft 



Boeing 

747-400 

(N=47) 

Douglas 

MD-11 

(N=39) 

Airbus 

A-320 

(N=46) 

Age 

42.6 

52.3 

43.8 

Total Hours 
Flying 

11,393 

16,154 

11,160 

Total Hours as 

Pilot-in- 

Command 

5,041 

9,045 

5,688 

Years of Formal 
Education 

16.3 

15.2 

15.8 


As will be clear from subsequent sections, these problems prevented us from 
drawing conclusions about some of the comparisons that we had intended to 
make. The aircraft differences we were able to report with confidence met the 
following criteria: 

1. The results for the different aircraft were in line with expectations. 

2. Supplementary analyses, based on an unconfounded subset of the data (e.g., 
only Boeing and Airbus first officers), supported the original analysis. 

Materials 

A package of materials was sent to each subject. It included a cover letter 
explaining the general intent of the survey, a one-page background questionnaire 
(see Appendix A), the survey, and an envelope in which to return the survey. 
The survey itself consisted of 34 pages, and included four sections with a total of 
94 questions (see Appendix A). 

Procedure 

U.S. commercial airlines were identified that operated the B747-400, MD-11, and 
A-320 aircraft. The airlines were called and a contact person (usually the Chief 
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Pilot) for each type of aircraft was requested. The Chief Pilots were called and 
informed about the study. Chief Pilots contacted Fleet Captains for each aircraft 
type, who posted flyers and/or messages where flight crews could read them. 
Equal numbers of Captain and First Officer volunteers were requested from each 
airline; the goal was to have 150 respondents, 25 Captains and 25 First Officers 
for each aircraft type, but as can be seen from Table 2, respondents for some 
airline and aircraft types were skewed in terms of number of Captains and First 
Officers , and the total number of subjects responding was less than desired. 
Volunteers signed up with Fleet Captains and a list of names and addresses was 
compiled and forwarded to one of the researchers, who then mailed the materials 
package to each subject on the list. 

Volunteers were requested to return the survey within a week and were paid 
$150 for completing the survey. One hundred sixty surveys were sent out and 
132 were returned (Surveys were accepted even if they were late). Subjects were 
asked to complete questions pertaining to specific automation components based 
on their current aircraft. They were given the name and number of one of the 
researchers so that they could call for clarification or explanation of any question 
they did not understand. Half a dozen calls were received, all requesting 
clarification of the automation pieces listed in questions 1-31. 

RESULTS 

The results for each of the major issues the questionnaire was designed to 
address will be discussed for the sample as a whole. Subsequently effects 
attributable to the aircraft currently flown will be explored. 

Philosophy 

In questions 32-41 and 46-48, subjects responded to thirteen questions designed 
to assess agreement with a human centered, as opposed to a fully autonomous 
viewpoint. A human-centeredness score was calculated for each subject by 
averaging the responses to the thirteen philosophy questions (Q32-41; 46-48). 
Before calculating the averages, the polarity of some of the questions was 
reversed, and the scores transformed accordingly, so that a five always 
represented the human-centered end of the scale. The polarity of the questions 
had been varied randomly, to avoid calling the subject’s attention to the 
dimension of interest. 

The average response to the twelve questions, across all 132 subjects, was 3.53. 
This score showed that subjects endorsed the human-centered view, but not as 
strongly as they could have. An examination of each of the separate philosophy 
questions showed that subjects responded on the human-centered side of the 
scale consistently for all but one question. (This exception will be discussed 
shortly). (See Figures la to lm showing the frequency of responses to each 
question at each scale point). 
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Q32 


Q35 



Figure la. Initial emphasis in training should be 
placed on learning about: 

the automation before the "basic airplane" before 

introducing the introducing the automation 

"basic airplane" 


Figure Id. It is likely that situational information 
about A/C systems given by automa- 
tion in the future: 

will always require can always be accepted 

confirmation as fad 


Q33 



Figure 1b. 1 would like to see the introduction 
of more automation that: 

assists the pilot in automatically solves 

problem solving problems 



Figure 1e. It is likely that procedural information, 
response recommendations and 
commands concerning A/C systems 
given by automation in the future: 

will always require can always be accepted 

confirmation as fact 


Q34 



more automation that: 

evaluates and advises automatically executes 

the flight crew on altema- alternative plans of action 
tive plans of actions 


Q37 



Figure If. It is likely that the pilot in the future will: 

not be responsible stilt be responsible for 

for flying the aircraft flying the aircraft 
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038 


041 



there will still be system 
failures that require devi- 
ations from standard 
procedures 


we can expect that every 
system failure wHI have 
a prescribed procedure 
to follow 



Figure 1i. In most cases, an automatic system 
should: 


prevent the aircraft from warn tie crew of errve- 

exceeding its performance lope exceedence, but 

envelope not restrict pilot control 


Q39 



1 2 3 4 5 

Figure 1h. The biggest obstacle to total flight 


pilot performance 


system design 


046 



Figure 1 k. Decision aids should emphasize: 

situation information response information 


040 



Figure 1i. In pilot training in the future, principles 
of navigation and aviation are likely to 
receive: 


less emphasis 


just as much emphasis 



one alternative 


a Ust of alternatives 
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048 



crew with: 

recommendations commands 


Question 37 (Figure If) elicited the strongest human-centered response. Subjects 
were nearly unanimous in proclaiming that the pilot will still be responsible for 
flying the aircraft in the future (mean =4.9). Not surprisingly, given this view, 
subjects also felt strongly on Q32 (Figure la) that beginning pilots should learn 
about the "basic airplane," before learning to use the automation (mean=4.05). 
They also showed a strong preference for automation that assists the pilot in 
problem solving, as compared to one that automatically solves problems (Q33, 
Figure lb), another central tenet of human-centered philosophy (mean, with 
reversed scale,=3.86). 

The only score to fall slightly to the full-automation side of the scale for the 
average subject was Q39 (Figure lh). This question asked whether the greatest 
obstacle to total flight safety was system design or human performance. The 
assumption was that proponents of the human-centered view would focus on the 
problem of poorly designed systems while proponents of the full-automation 
view would point their finger at human foibles. Interestingly subjects proved to 
be more sympathetic to the full-automation view on this question than on any of 
the others, responding on the "human performance" half of the scale (mean=2.61, 
reversed scale). The respondents may have been influenced by the statistic that 
"pilot error" is responsible for two thirds of all aircraft accidents (Regal & 
Braune, 1992). Another interesting finding was that on the question concerning 
envelope protection (Q41, Figure lj), the B-747 and MD-11 pilots responded on 
the human-centered side, while the A-320 pilots responded on the full- 
automation side. This result will be discussed in the section on Aircraft 
Differences. 

These results provide support in the pilot community for the human-centered 
design philosophy that has been favored by researchers. In constructing the 
scale, effort was made to avoid making one of the poles sound like the "right 
answer." The fact that only one of the questions was close to the ceiling (Q, 37), 
lends credence to the measurement technique and suggests that the questions 
received careful consideration. 
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Good/Tr ustworthv Automation 

The average ranking was calculated for each of the eleven attributes for good 
automation (Q51) and then for trustworthy automation (Q52). The attribute that 
received the most favorable average ranking in each case was then ranked #1 the 

next best attribute #2, and so forth. The rank orders obtained in this wav' are 
shown m Table 4. y 


Table 4 

Rank Ordering of Attributes for Good and Trustworthy Automation 


Q51-52 


Attribute 

Rank for Good 
Automation (Q51) 

Rank for Trustworthy 
Automation (Q52) 

Dependable 

1 

1 ~~ ~j 

Predictable 

- y ~ 

2 

Simple 

3 

r ~ 

Comprehensible 

j “ 4 

3 

Informative 

5 

i ~~5 

Subordinate 

6 

r~ 8 

Flexible 

7 

10 

Error-resistant 

— | 

7 

Accountable 

9 “ ~ 

6 

Adaptable 

To [ 

11 

Error-tolerant 

li 

9 


Nfiifi- The ranks indicated are a rank ordering of the average ranks given for the attributes. 

As can be seen in the table, the results for the two questions were highly 
correlated (Spearman's R= 86, pc.001). Although, Q51 had stressed safety and 
economy, while Q52 had emphasized pilot acceptance and use, subjects failed to 
make a distinction between "good" and "trustworthy" automation. Striking in 
both cases was the clear "back to basics" response. An examination of the first 
five ranks showed that pilots want automation that works (i.e., is dependable) 
and that they can understand (i.e., is simple, predictable, and comprehensible). 
All other features were considered, by their lower ranks, to be secondary. The 
message seems to be: Don't try anything fancy (i.e., flexible, adaptable, 
accountable, error-tolerant or error-resistant), just give me something I can use. 

Billings (1991) has suggested that certain of the attributes on the list oppose 
certain others, in the sense that emphasizing one often forces a de-emphasis on 
the other. The following attribute pairs, he believes, bear such a relationship: 
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• Accountable — Subordinate 

• Predictable — Adaptable 

• Comprehensible — Flexible 

• Dependable — Informative 

• Error-resistant— Error-tolerant 

The present results suggest the following prioritization for these pairs. With the 
exception of the first pair (accountable-subordinate), for which rankings were 
inconsistent across Q51 and Q 52, the rankings showed a clear preference for the 
left hand members of each pair. 

With respect to the problem of human error, the rankings suggest that pilots 
want to prevent error from happening in the first place rather than trusting a 
machine to correct it. Towards that end, their first desire is for automation that is 
easy to use (ranks 1-5), their second is for error-resistant systems that wiU check 
for typical kinds of mistakes (rank 7 or 8) and their last is for automation that will 
try to help out after an error has occurred (rank 9 or 11). 

In sum, the rankings show a preference for simple, easy to use systems over truly 
leading edge technologies. They lend further support to the human-centered 
attitudes of the pilots revealed in the philosophy questions. 


Physical and Mental Workload 

In Question 55, respondents were asked to rate workload for glass cockpits m 
comparison with conventional cockpits. Responses were made on a five point 
scale where 1 was "much lower than in conventional cockpits, 3 was about the 
same," and 5 was "much higher." The average physical and mental workload 
ratings obtained for each phase of a flight are shown in Figure 2. These averages 
are based on 111 out of the 132 subjects. Four subjects had to be eliminated for 
failure to respond to all parts of the question. The additional reduction in sample 
size was necessary to equalize the numbers from each of the three aircraft groups 
so that a mixed-factors analysis of variance could be carried out, with aircraft as a 
between-subjects factor (Results of the aircraft analysis will be discussed m a 
later section) Subjects were retained in the order in which they had responded 


to the survey. 
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Ei gUJg 2. Physical and mental workload for different flight phases. 

The results support the conventional wisdom that automation has reduced 
physical workload more than it has mental workload. Regarding physical 
workload, inspection of the portion of the curve that is below 3.0 ("about the 
same as in conventional cockpits") shows that physical workload was rated as 
being lower than in the conventional cockpit for all flight phases except two: pre- 
flight (4.04), which was substantially higher and taxi (3.14), which was about the 
same. The physical aspect of punching buttons into the computer and talking to 
ATC in these phases may be responsible for the high physical workload ratings 
for these phases. While these actions are not physically exhausting, in the usual 
sense of high physical workload, they do entail being physically busy. The 
biggest reduction in physical demands can be seen in the cruise phase (1 69) 

presumably reflecting the widespread use of flight control and navigation 
automation. ° 


Mental workload ratings, showed the same general ups and downs across phases 
as physical workload. However, the difference in workload in glass cockpits in 
relation to conventional cockpits was more pronounced for physical workload 
than for mental workload. 


These observations were supported by statistical analysis. A two-way repeated 
measures analysis of variance, with workload type (physical vs. mental 
workload) and flight phase as within-subject variables revealed a main effect of 
Fn ' ,10,= 30.57, pc.OOOl, a main effect of flight phase 
M6,66(J)-1215.6, p<.0001, and a significant interaction between workload tvoe 
and flight phase (F(6,660)=5.671, p<.0001. 
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Individual comparisons were carried out by a Tukey test (significance was 
assessed at the .05 level for all Tukey tests in this report). The results revealed 
that the rating of mental workload differences between glass and convention^ 
cockpits were significantly different than the rating for physical workload 
differences for the cruise (p<.05) and approach (p<.05) phases and just missed 
being significantly different for landing (pc.lO). 

The Tukey test also showed the following pattern of change over flight phases: 
Ratings of both physical and mental workload differences between glass and 
conventional cockpits changed significantly from pre-flight to taw (p<05) 
from taxi to take-off (p<.05). Physical, but not mental workload differences, 
changed from climb to cruise (p< 05), and both types of workload difference 
ratings changed from cruise to approach (p<.05). 

In summary, the results showed that pilots feel that mental workload has not 
been reduced in the automated cockpit to the same extent that physical workload 
has. Both physical and mental workload were rated as markedly higher in glass 
cockpits than conventional cockpits for pre-flight, and slightly higher for taxi. 
Mental workload was rated as slightly higher or the same in glass cockpits for 

approach and landing. 

levels of Automation: Promise and Concerns 

In questions 78-80, subjects indicated how much they agreed or disagreed with 
claims about different levels of automation. First, they rated all the possible 
advantages, or promises, of each level (part ’a*) and then the possible 
disadvantages, or concerns (part b ). 

An overall "promise" and an overall "concerns" measure were obtained for each 
of the three levels of automation of interest by averaging the responses to each 
item in Q78-80a, and Q78-80b, respectively for each subject. The average 
measures, across subjects, are shown in Figures 3a and 3b, respectively. The 
number of subjects included in the analysis, after eliminating the few ^non- 
responders and equalizing the aircraft groups, was 114 and 111 out of 132 for 
the" promises" and "concerns" analyses, respectively. 

The overall promise rating (see Figure 3a) was significantly lower ^ to pilot 
unassisted (2.68) than for the shared or fully autonomous levels, F(2^26)-9L813, 
d< 001 The shared and autonomous levels were rated identically overall (both 
3.79). An examination of Table 5a shows that pilots felt that the autonomous 
level has the most promise to "alleviate fatigue," "reduce worUoad, provide 
more precise data," "increase safety," and "increase airline cost effecfaveness but 
could not match the shared level for "keep me involved, keep me mformed, 
improve my performance," and "improve my situation awareness. Neither level 
could top the unassisted level for "keep me involved." 
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Concern ratings showed a dear differentiation of levels (see Figure 3b). Overall 
concern grew from 2.3 to 3.29, as automation increased, which was a significant 
difference, F (2,220)=96.001, p<.0001. A Tukey test showed that each of the levels 
differed significantly from the others. These results were highly stable: 
Inspection of individual items (see Table 5b) showed that all but four of the items 
increased regularly across levels. 

Major concerns at the fully automated level were increased head-down time 
(4.05), complacency (3.95), and degradation of pilot skills (3.90). These items, 
which have to do with the pilot’s ability to "stay in the loop," are related to the 
fear that full automation will not "keep me involved" expressed earlier in the 
promises section. Fourteen out of the eighteen items on the list received scores 
above 3.0, indicating they were of moderate to high concern at the fully 
automated level. The four that were of lower than moderate concern were 
temperamental devices," "difficulty in learning to operate," "need for new 
skills," and "difficulty in detecting system errors." 



bility 

Level of Automation 


Figure 3a. The promise of different levels of automation. 
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Q78-80b 



Unassisted 


Responsi- 

bility 

Automation Lovel 


Autonomous 


Figure 3b. Concern about different levels of automation. 


Table 5a 

Average Promise Rating at Each Level of Automation 


Q78a-80a 


Promise 

Pilot Unassisted 

Shared Control 

Autonomous 

keep me involved 

4.24 

4.14 

3.15 

keep me informed 

3.18 

4.21 

3.99 j 

improve my 
performance 

2.87 

3.89 

3.56 

improve my 

situation 

awareness 

2.98 

3.96 

3.78 

alleviate fatigue 

2.03 

3.47 

3.65 

reduce workload 

1.88 

3.42 

4.08 

provide more 
precise data 

2.32 

3.71 

4.14 

IM,« 

2.25 

3.82 

3.90 

increase airline 
cost effectiveness 

2.42 

3.58 

3.68 


Note . The rating scale was: 5 = agree 1= disagree "that automation at this level holds the promise 


to..." 
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Table 5b 

Average Concern Rating at Each Level of Automation 

Q78b-80b 


Disadvantage 

Pilot 

Unassisted 

Shared Control 

Autonomous 

temperamental devices 

1.85 

] 2.48 

2.76 


1 2.15 

2.78 

3.13 

losing sight of the 
raw data 

1.90 

2.77 

3.34 

difficulty in learning 
to operate 

2.05 

2.54 

2.95 

data entry errors 

2.33 

3.12 

| 3.32 

software engineering 
errors 

2.09 

3.11 

3.43 

need for new skills 

2.06 

2.62 

2.78 

unforeseen and 
unintended negative 
consequences 

2.83 

2.82 

3.10 

workload extremes 
(high and low) 

3.83 

3.10 

3.32 

increase in the no. of 
alerting signals 

2.46 

3.11 

3.40 

loss of situation 
awareness 

2.94 

2.65 

3.16 

need to work around 
automation in unusual 
circumstances 

2.06 

2.94 

3.34 

increased head-down 
time 

2.67 

3.54 

4.05 

difficulty recovering 
from automation failure 

1.94 

2.97 

3.46 

degradation of pilot skills 

1.85 

3.11 

3.90 

difficulty in detecting 
system errors 

3.12 

2.73 

2.71 

reluctance of crew to 
take over from automation 

1.78 

2.60 

3.12 

complacency 

2.30 

3.13 

3.95 


Nfilg. The rating scale was: 5 = high concern 1 = low concern "that automation at this level could 
lead to ...” 
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For shared performance, only seven out of the eighteen items were rated higher 
on average than "3," indicating more than moderate concern. The items that 
dropped from the high concern half of the scale to the low half between fully 
autonomous operation and shared performance were: display complexity, losing 
sight of raw data, unforeseen and unintended negative consequences, loss of 
situation awareness, need to work around the automatics in unusual 
circumstances, difficulty in recovering from an automation failure, and 
reluctance of crew to take over from automatics. 

Although pilots generally had fewer concerns about flying unassisted than about 
flying with automation, they did see clear advantages to automation. Two 
concerns, workload extremes (3.83) and difficulty in detecting system errors 
(3.12), were seen as concerns at the unassisted level that were alleviated by the 
introduction of automation. 

In summary, subjects showed, through their concerns and promise ratings, that 
they have no trouble thinking about three different levels of automation. In 
terms of costs and benefits, the intermediate level seemed to be the most 
advantageous, because it showed as much promise as the fully automated level 

with fewer concerns. 

THpal Level of Automation for Di fferent Situations 

In a series of questions subjects were asked to rate the maximum level (Q56-77) 
of automation on their current aircraft (see part 'a') and the level that they would 
ideally use (see 'b' part ). The questions were asked for five attributes of the 
situation that could affect the actual and ideal levels of automation: flight 
condition (i.e., normal, non-normal, emergency) (Q56-58); phase of flight (Q59- 
66); mission function (i.e., flight control, navigation, communication, systems 
management, and task management) (Q67-71); human information processing 
task (i.e., monitoring, processing, responding) (Q72-74); and workload (i.e., high, 
medium, low) (75-77). 

A measure of the subject's satisfaction with the level of automation in his current 
aircraft, with respect to the ideal, was calculated by subtracting the ideal rating 
from the actual rating for each question. The "actual minus ideal rating captures 
the subject’s satisfaction in the following way: A positive rating means that a 
higher level of automation is available than pilots ideally would use, a negative 
rating means that the highest level available is lower than they ideally would 
use, and a zero means that the highest level available is the same as the ideal 
level they would use. A negative rating indicates that perhaps higher 
automation levels should be considered for that particular situation. 

The "actual minus ideal" ratings for each of the five situation attributes are shown 
in Figures 4a-4e. For these analyses, the sample was again reduced to 111 out of 
132 to achieve equal numbers in the three aircraft groups. A one-way, repeated 
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measures analy s is °f variance, with category as the factor (e.g., high, medium, 
Mid low for the situation attribute labeled "workload,") was carried out on the 
actual minus ideal scores for each of the situation attributes. The results 
showed a significant main effect of category (p<.0003 for human information 
processing task, and pc.OOOl for all other situation attributes). In other words, the 
different perspectives afforded by each of the situation attributes helped subjects 
to identify situation categories for which they could use higher levels of 

suffident° n C ° ntraSt to those for which the available automation level was 

The most interesting categories in each analysis were those for which pilots 
indicated a level of automation that departed from the ideal. For every bar in 
igures 4a-4e, a t-test for repeated measures was conducted to determine if the 
actual level (part 'a') was significantly below or above the ideal ( part 'b') To 
correct for multiple comparisons, the probability level for the t tests in each 
classification scheme was multiplied by the number of t tests conducted. 

The following categories were found to be significantly above the ideal in the 
IT?, of a . U . t ° ma ^on available to aid pilots (p<.05, adjusted): normal flight (see 
flight conditions Figure 4a), climb, cruise, descent, and landing (see flight phase 
Figure 4b); flight control (see mission function, Figure 4c). This finding does not 
m f ai \ that the higher levels of automation should be eliminated for these 
situation categories, but it does indicate that lower levels should be available for 
pilot use. It also has implications for the likelihood that pilots will resist using 

lugher levels of automation in some situations, and should be taken into account 
m training and operations. 



Normal Non-Normal Emergency 

Flight Conditions 

Ei gure 4a. Satisfaction with level of automation for different flight 
conditions. A positive rating means that the automation 
level available is higher than the respondent's ideal level. 
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Figure 4b. 


Satisfaction with level of automation for different flight 
phases. A positive rating means that the automation level 
available is higher than the respondent’s ideal level. 



Fi<mre 4c. Satisfaction with level of automation for different mission 
* functions. A positive rating means that the automation 

level available is higher than the respondent’s ideal level. 
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Actual • Ideal Rating 


Q72-74 



Cognitive Task 

Figure 4d. Satisfaction with level of automation for different cog- 
nitive tasks. A positive rating means that the automation 
level available is higher than the respondent's ideal level. 



Figure 4e. Satisfaction with level of automation for different 

workloads. A positive rating means that the automation 
level available is higher than the respondent's ideal level. 
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The following categories were found to be significantly below the ideal in the 
level of automation available to aid pilots (p <.05, adjusted): preflight and taxi 
(see flight phase, Figure 4b); communication and task management (see mission 
function, Figure 4c), responding, defined as performing goal-oriented actions (see 
human information processing task Figure 4d); high workload (see workload, Figure 
4e). This finding, of the need for more automation to deal with pre-flight and 
taxi, communication and task management, and high workload situations, 
reinforces the message of earlier sections — a plea for attention to the crew’s 
mental workload. 

Amount of Automation 

In a series of questions (Q81-85), subjects were asked to rate the amount of 
different types of automation on their current aircraft (part 'a') and then to 
indicate whether ideally, they would want less, the same amount, or more 
automation than they currently have (part ’b’). The average ratings for the two 
parts of the questions are shown in Figures 5a and 5b, respectively. 

The results of part 'a' (see Figure 5a) showed that subjects felt that their aircraft 
had at least a "moderate" amount of automation in all categories. The pilots felt 
they had the largest amounts of automation for aircraft control (mean of 4.34) 
and systems control (4.18.). Information automation, defined as automation that 
informs the pilots about the aircraft and systems states, operations, procedures, 
regulations, and location of the aircraft and relevant entities in the environment, 
was also considered prevalent (3.94). Pilots felt they had a lower amount of 
protective automation (3.52). Decision automation, defined as automation that 
aids the pilot in selecting alternatives and making choices, came out lowest (3.11). 

An F test for repeated measures indicated that the overall difference between 
automation categories was significant, F(4, 440)=59.169, p<.0001. A Tukey test 
indicated that each of the ratings for the different automation categories differed 
from the others (p<.05), with the exception of one pair, aircraft control and 
systems control, which were statistically indistinguishable (p>.05). 



Minimal 1 l ■■ 1 * * - I ■ I 

Control Systems Into Decision Protec- 

tion 

Automation Type 

Figure 5a. Amount of different types of automation in aircraft 
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Figure 5bi Desire for less, same amount, or more automation* 


The results for part 'b/ in which pilots indicated the ideal amount of automation 
with respect to their current aircraft are shown in Figure 5b. On this scale, a 3.0 
indicates a belief that current levels are ideal, a score of more than 3.0 means that 
more automation is desired and a score below 3.0 means that less is desired The 
average ratings were all slightly above 3.0, indicating that pilots would prefer, 
ideally, to have slightly more automation than they currently have in all 
categories. The two categories that came closest to the ideal were aircraft control 
(3.3), followed by protection (3.4) automation. Although pilots indicated in part 
'a' that they did not have a large amount of protective automation on their 
aircraft, they evidently felt that the amount was dose to ideal. The biggest needs 
were for more information (3.7), dedsion (3.6), and systems (3.6) automation. 


An analysis of variance for repeated measures showed a significant effect of 
category, F(4,440) =8.457, pc. 0001. A Tukey test confirmed that the need for 
information and dedsion automation was significantly greater (pc.05) than the 
need for more control or protective automation. The need for systems 
automation was also significantly greater (pc.05) than the need for more control 
automation. 


Phenomenological Experiences 

In questions 1-26 subjects rated different automation systems on seven 
dimensions (parts 'a' through ’g'). Questions 27-31 were similar, except that 
subjects rated different cockpit alerts on six different dimensions (parts 'a' 
through T). Tables B1-B13 in Appendix B show the order in which the 
components fell on each of the dimensions, respectively. All 132 subjects were 
induded in this analysis, but subjects were not required to answer all questions. 
They were requested to leave a blank if they were not familiar with the 
component. Overall, subjects left 6.8% of the components unrated (data showing 
which components were omitted will be presented in a subsequent section 
concerned with aircraft differences). 
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Ratines were averaged across subjects and automation components for each 
dimension queried. Before the averages were calculated, the rating scales for 
some of the questions (parts a, c, d, and f of Ql-26) were reversed from the way 
they appeared on the questionnaire, so that a ’5’ always indicated the desirab 

end of the dimension. 

The average rating across all components and subjects is shown in Table 6 for 
each of thf dimensions. It is clear from these data that subjects were favorably 
inclined towards the automation systems on their flight deck, though they had 
some reservations. On questions 1-26, they found the automation components to 
^unobtrusive (4.00), predictable (3.97), extremely helpful for redu^g woridoad 
(3 82) and they were inclined to use them whenever appropriate W- 3 9). On th 
other hand, they were close to the midpoint when it came to the feehng that they 
were controlling the flight rather than managing the automation (3.25, where jj 
high controlling), and the feeling that they were focusing on the flight rather than 
on the automation (3.61, where ’5’ = attention to flight). 

For questions 27-31, subjects clearly trusted the alerts (4.27) and found them 
compelling (4.87, see Table 6). They gave lower ratings, however, to the ease of 
returning to tasks after an alert (3.50), degree to which the alert can be responded 
to without thought (3.52), and immediacy of knowing what the problem * 065). 
While these responses were on the positive half of the scale, they were 

the borderline. 

To assess whether arty of the separate dimensions might have a con^on basis 
from the pilots' perspective, correlations were calculated for each of the 
dimensions in Ql-26 and Q27-31, respectively, paired with each of the others. 
The two correlation matrices are presented in Tables 7a and 7b. 

For Ql-26, all but one correlation were significant (see Table 7a). The three 
highest correlations were for: 1) the two dimensions describing the automation 
(unobtrusiveness V and predictably V. R-.7050. P< 001),2 the tw^tang 
the pilot (situation awareness ’b' and workload reduction f, R-.6698, p<.OOU 
and 3) the two describing the interaction between pilot and automation attention 
on flight/automation ’a' and feeling of controlling/managing d , ( R -f254, 
p< 001). These three clusters, while not completely independent, provi 
useful structure for thinking about automation experiences. 
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Table 6 

Average Rating of Automation Components on Each Dimension 
Ql-26 and Q27-31 


Questions 





_| Dimension 

' attention on tlightVautomation 


understanding of the big picture 


unobtrusiveness 


controllingVmanaging 


predictability 


workload reduction 


frequency of use 


trust in alert 


knowing what the problem is 
ability to respond without thought 


ease of returning to task after alert 


l scarcity of alert occurrences 


Average Across 
Components 


3.61 


3.79 


4.00 


3.25 


3.97 


3.82 


4.39 


4.87 


4.27 


3.65 


3.52 


3.50 


3.65 


Nfite. Rating scales for some dimensions were reversed from the way they appeared in the 
that 5 * the favorable end of the continuum for all di^i™ T* * u^ to 
md, cate the favorable end where it is not obvious from the dimension^ ‘° 


Table 7a 

Correlation of the a-g Components for Q1-26+ 



+ Seven component scores (averaged across Ql-26) were entered for each subject 

“ ay " appeared in “* *> V would 

Significant at the .05 level. ""Significant at the .01 level. 
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Table 7b 

Correlation of the a-f Components for Q27-31 + 



+ Six component scores (averaged across Q27-31) were entered for each subject. 

•Significant at the .05 level. 

^Significant at the .01 level. 


The high correlation found between the two questions intended to explore the 
direct/mediated experience (attention on flight/automation, ° 

rrmtrnllinc /supervising) suggests that this distinction has psychological validity. 
^b^^^^^d P B4 / in U Appendix B show the components listed in order of the 
Jatin7s Received on these dimensions. Components at the top of the list 
wererated highest on allowing attention to the flight (or a feeling o con g 
the flight), while those at the bottom were highest on directing attention t 
automation (or a feeling of managing the automation). 

An examination of the lists in Tables B1 and B4 in Appendix B substantiates that 
^ 'a' and ’d' dimensions were highly correlated. In both cases, the ! types o 
automation at the top and bottom differed strikingly from each other. Atthetop 
nf both lists representing the extreme direct experience (attention to flight a 
feehm; of controlling the 8 flight) are: the hydraulic amplification °* in P u *' 
emund sailer, fly by-wire controls, and automatic rudder. As would be 
expected, ^he ratings confirm that these controls require little attention and do 
notdetract from the flying experience. At the bottom, ^ th 

mediated experience (attention to the automation, feeling of managingthe 
rutomattn) P by contrast, are: FMC-flight planning FMC-aut 
TNAV and VNAV, ACARS, and subsystem schematics, as well as the auto 
todl wr*e exception of auto landing, these comgmentt ^eqmre 
»Ce 8 x pilot input as previous studies have suggested (Sarter & Woods, 1994), 
or are complicated enough to distract attention away from the flight. 
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The inclusion of auto landing in this group of highly interactive systems rather 
than with the other flight control systems at the top of the list is revealing. It 
suggests that landing with this system does not free the pilot from doing the job 
at least not mentally. The lack of familiarity with the system and the fact that this' 
flight phase is extremely critical may be factors contributing to this feeling. 

pie correlation between situation awareness and workload reduction also is of 
interest. Note that the correlation is a positive one: Components listed as 
resulting in higher situation awareness generally were rated high in workload 
reduction (see Tables B2 and B6 in Appendix B). This finding is very 
encouraging since one might expect situation awareness to incur a workload 
penalty— that is, it takes effort and attention to develop and maintain situation 
awareness. The fact that automation components generally seem to increase 
situation awareness without increasing workload is a somewhat unexpected 
benefit. There are notable exceptions however. The FMC and TCAS traffic 
display were rated as providing high situation awareness, but also were rated as 
high in workload (see Tables B2 and B6 in Appendix B). The improved situation 
awareness with these components does come with a penalty. The automatic 
rudder and hydraulic amplification of inputs were rated low in workload but 
also low in situation awareness. These represent the classic case of automating 
functions that take the pilot out of the loop which results in their being less 
involved and informed than if the function were not automated. 

For Questions 27-31, most of the correlations between dimensions were 
significant (see Table 7b). However, there was no evidence for any higher-order 
dimensions. It is interesting to note the strong negative relationship between 
bust and frequency of alert (R-.3316); it is very likely higher frequency means 
higher false alarm rates, which obviously reduces trust in the alert. The alerts 
that are trusted the least (master caution and warning and TCAS traffic 
advisory), are also the ones that require the most thought in responding (see 
Appendix B, Tables B9 and Bll). 6 

Aircraft Differences 

The effect of flight deck experiences on pilot attitudes and opinions was 
examined by comparing survey responses of the pilots in the three aircraft 
groups (Boeing 747-400, Douglas MD-11, and Airbus A-320). These flight decks 
are thought to vary along a dimension of increasing technology, from Boeing, to 
Douglas, to Airbus. Evidence in support of this difference was sought by 
examining the number of components in questions 1-26 that subjects were able to 
rate. Subjects were instructed to leave a question blank if the aircraft that they 
currently fly did not have that automation component. Because the list was 
instructed to be a representative inventory of the types of automation systems 
found on advanced flight decks, the number of blanks serves as at least a rough 
measure of the extent of the automation on the different aircraft. The average 
number of blanks was : Boeing 2.8, Douglas 1.4, Airbus 1.1. An analysis of 
variance showed a significant effect of aircraft, F(2, 129)=11.258, pc.OOOl. A 
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Tukey test revealed that Boeing had significantly more blanks than either Airbus 
or Douglas. However, the latter two did not differ from each other (Tukey 
p>.05). Using blanks as a measure, we can at least be confident that Boeing is ess 

automated than the other two groups. 


PhilosoDhv. There were no overall group differences in responses to the 
philosophy questions (Q32-42; 46-48), F(2,129)<1, p>.05. The average human- 
centeredness scores were: Boeing 3.55 Douglas 3.54, Airbus 3.49. However, 
examination of individual questions revealed a significant difference on queshon 
41, which concerned attitudes towards envelope protection, F(2,129)-7.728. 
p<.0007. A Tukey test indicated that the mean for the Airbus pilots (2.67) was 
significantly (p<.05) less human-centered than the mean for either the Boeing 
(3 45 ) or Douglas pilots (3.59), who did not differ from each other (p>.05). On 
this question, the Airbus pilots were the only ones to favor an envelope system 
that actually prevented envelope exceedence as opposed to just providing an 
advisory. This difference is interesting because it reflects the envelope features 
available on the respective flight decks. 


Good /Trustworthy Automation . There were no obvious group differences in 
rankings of the attributes for good and trustworthy automation in questions 51 
and 52. The average ranking for each of the attributes was similar across groups. 


Physical and Mpntal Workload. Analysis of the workload ratings in question 55 
produced one of the two results (see next section) in which the Douglas pilots 
appeared to differ from the others. In this case, there was a significant 
interaction between aircraft type and flight phase, F (12, 648)-2.639, p <.002, with 
the Douglas pilots reporting a significantly higher overall workload than the 
other pilots for two flight phases, approach and landing (Tukey, p<.05). These 
results are uninterpretable, however, because of the confounding with 
background. They may reflect the conservatism of an older and more 
experienced group of pilots, rather than a real aircraft difference. 

levels of Automation: Promis e and Concerns. Analysis of the promise ratings 
for the three different levels of automation in question 78 showed no airo-att 
differences. The concern ratings, however, were similar to the workload ratings 
discussed above in showing a higher overall concern from the Douglas pilots 
than from the others at every level, F(2, 108)=6.922, p<.002; Tukey<.05). Agam, 
these results may reflect a conservative bent rather than real aircraft differences. 


Ideal Level gf Automation for Diffe rent Situations. The analysis of actual versus 
ideal levels of automation across different situation attributes, encompassing 
questions 56-75 yielded clear and robust group differences. Figures 6 a- 6 e show 
the satisfaction measures (actual minus ideal rating) ) for the categories of eadi of 
the situation attributes. It is clear from the figures that in every case in which 
there was a negative bar (indicating a maximum available level of automation 
that was less than ideal), pilots of the Boeing 747-400 responded more negatively 
than did the other two groups. These trends were confirmed by conducting an 
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analysis of variance for each situation attribute with aircraft as a between- 
subjects variable and category and judgment (actual, ideal) as within-subjects 
variables. In all but one case (Q72-74), a significant three-way interaction was 
found, confirming that for those categories that were rated as not ideal, Boeing 
pilots tended to perceive their available level of automation as being farther 
below the ideal than did the others. 


Q56-58 
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H Boeing 
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H Airbus 


Normal Non- Emer- 
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Flight Conditions 


Figure fat Satisfaction with level of automation for different flight 
conditions; aircraft differences. 


15 Q59-66 



Q. 


Fllght Phase 

Figure 6t>, Satisfaction with level of automation for different flight phases; 
aircraft differences. 
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Fi gure 6c. Satisfaction with level of automation for different mission 
functions; aircraft differences. 
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Fi gure 6d. Satisfaction with level of automation for different cognitive 
tasks; aircraft differences. 
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Figure Satisfaction with level of automation for different workloads* 
aircraft differences. ' 


The details of these interactions are: 


Q56-58, (see Figure 6a). The closer the situation is to an emergency, the more 

^, € ?^ ially If" 8 pilotS ' could use a hi S her level of automation, 
F(4,216)=5.272, p<.0005. 


Q59-66, (see Figure 6b). A higher level of automation is desired, especially by 
Boeing pilots, at the beginning (pre-flight, taxi) and very end of the flight, (taxi & 

park, but not landing), F(14,756)=2.366,p<.004. 


Q67-71 (see Figure 6c). A higher level of automation would be helpful for 
communication, system management, and task management, especially for 
Boeing pilots, F(8,432)=2.369, p<.02. K y 


Q72-74 (see Figure 6d). This question failed to show a three-way interaction. 
However, a significant two-way interaction, between aircraft and judgment 
(actual, ideal) confirmed that Boeing pilots rated the maximum available level of 

the ideal than lhe others for c ° 8niUve tasks m s enera1 / 


Q75-77 (see Figure 6e). The higher the workload, the more pilots, especially in 
the Boeing group, feel the need for a higher level of automation F(4,216)=2.621 


To be sure that these group differences in satisfaction with automation levels 
were not an artifact of certain uncontrolled variables, a reanalysis of the data was 
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carried out using a subset of the data. To eliminate the 

seat and age, only the date from the Boeing and Airbus Fust Officers was yarned 
out (N=23 per group). (A similar analysis for Captains was notpossible because 
of insufficient numbers in the Boeing sample, see Table 2 The restrits ofthe 
First Officer analysis supported the results obtained for the larger sample. Three 
out of the four three-way interactions that had been ™ 
significant (p<.02); the one that did not reach significance (Q56-58) showed a 
trend in theright direction. In addition the two-way interaction m Q72-74 
remained significant (p<.002). One possible explanation of the consistently large 
negative gaps in the available minus ideal level of automation ratings for tit 
Boeing pilots is the lower available level of automation reported by these pilots. 

Amount of Automation . Ratings of the amount of automation m ^eir current 
aircraft (Q81-85a) for the three pilot groups are shown m Figure 7a. « is clear 
from the figure that the Boeing pilots felt they had lower amounts of automation, 
than did the other pilots. This difference was most pronounced for _^tecbve 
automation, where A-320 pilots reported the most 

reported the least. The results reinforce the original assumption that the Boeing 
747-400 is less automated than the other two aircraft and suggest that the Airbus 
A-320 is more automated than the MD-11 at least in the area o pro ec ve 

automation. 

An analysis of variance with aircraft type as a between-subjects variable and 
automation type as a within-subjects variable supported these trends by showing 
a S^n effect of aircSft group, F(2,108) =11-31, P< 0001 a sig^ifican 
maiTeffect of automation type, F(4,432)=62.44, ,£000^ 
interaction between aircraft group and automation type, F(8,432) . ,] P<- 

A Tukey test revealed no significant differences between groups 
systems automation. For information and decision automation, 747-400 pi ot 
ratings differed significantly from those of MD-11 pilots (p<.05) and marginal y 
significantly (p<.10) from those of A-320 pilots. All three aircraft ipoups differed 
significantly from each other in the perceived amount of protective automation 

(p<.05). 

Ratines of the ideal amount with respect to the current amount (Q81-85b) are 
S Figure 7b. It is evident from the figure that Boeing pilots expressed a 
greater desire for more automation than did the other two groups, especially m 
the categories of systems, information, and decision automation. 

An analysis of variance confirmed these trends by showing a marginally 
^Lfica^t m^n effect of aircraft type (F, 2,108)=2.777, p<.10, a significant main 
effelt of automation type (F4,432)=8.644, p<.W01 «d • stg^ftcant 
between aircraft and automation type, F(8, 432)-2.214, p<^3. A Ttuey 
revealed only one significant difference between groups: 7 * 7 - 401 pilots d«ir e 
for more information automation was sigmficantlygreatert^t^tofMWT 
pilots (p<.05). In addition, a trend toward greater desire on thep ar to 747-400 
pilots « found for systems automation, where the difference between 747 
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pilots and MD-11 pilots was marginally significant (pc.10), and for decision 
automation, where the difference between 747-400 pilots and A-320 pilots was 
marginally significant (pc.10). 



o. 


Automation Type 

Figure 7a. Amount of different types of automation; aircraft differences. 
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Figure 7b. Desire for less, same amount, or more automation; aircraft 
differences. 
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An analysis of variance with data from First Officers only (23 Boeing and 23 
Airbus) supported the conclusions about group differences in ratings of the 
amount of automation on the current aircraft (part ’a’) but not of the desire for a 
greater or lesser amount of automation (part 'b'). In part 'a/ the main effects of 
aircraft and automation type were significant (p<.0002), as was the interaction 
between aircraft and automation type (p<.02). Only the main effect of 
automation type was significant (pc.02) in part 'b.' 

Phenomenological E xperiences 

No group differences were apparent in the ranking of the components in 
questions 1-31 on any of the scales. 

DISCUSSION 


Philosophy 

Pilots, as expected, generally endorse human-centered positions about use of 
automation. The notable exceptions were the opinion that pilots, and not design, 
are the biggest obstacle to flight safety and the opinion among A-320 pilots that 
automatic systems (envelope protection) should prevent the aircraft from 
exceeding its performance envelope rather than providing alerts and allowing 
the pilot to override soft limits. This position among A-320 pilots may be an 
indication that pilots’ philosophy is influenced to some extent by their training 
and aircraft design philosophy. 

Good /Trustworthy Automation 

Pilot rankings of the importance of eleven automation features or attributes were 
highly correlated for "good" and for "trustworthy" automation, indicating either 
that pilots do not distinguish between good and trustworthy, or that the same 
automation attributes are important for both. In both cases, pilots indicated that 
they want automation to be dependable, predictable, simple and comprehensible. 
It is less important for it to be flexible, error-resistant, accountable, adaptable, 
and error-tolerant. The message from pilots to designers seems to be: Don't try 
to be fancy, just make sure it I can use it and it works. 

Physical & Mental Workload 

Pilots' ratings of differences in mental and physical workload between glass and 
conventional cockpits were phase dependent, with higher relative workloads in 
glass cockpits at either end of the flight than in the middle. Workload was rated 
as being much higher than it had been in conventional cockpits during preflight 
and similar to how it used to be for taxi and approach. Reductions in workload 
with glass cockpits were found for take-off, climb, cruise and landing, where 
ratings were lower than for conventional cockpits (rating < 3.0). These results 
suggest that efforts to reduce workload during pre-flight, when activities such as 
system initialization and data entry must be performed, and during taxi. 
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available. These situations included- nomal^gM^Infcf aUt ° ma * ion than is 
la ndin g phases of fliahf- an^ n- us , r cruise, descent and 

that tts°° n f . ^ design implication is 

operationally sanctioned even if WgLAlTerfst 8 h^ouW “h 1 "* 

current aircraft do make lower levels!, auton^^ai^rUti:^' 

Amount of Aut omation 

deSl^'MD-n and ^320 Dilots° f d ,he ° f automation ** were 

automaHon in each “4ore ^ dfd “ havin « — 

between theMD-11 anri a 47-400 pilots. The only difference 

ESlpSSaHS 
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Phpnomenological Ex periences 

Automation components on current aircraft vary widely on the various 
dimensions that were explored in terms of obtrusiveness, workload, situation 
awareness, predictability, and sense of attending to and performing 
underlying function versus managing the automation. As expected, compone 
reauirimAignificant input or monitoring, such as the flight management 
computer and ACARS are more obtrusive, workload intensive, and result m the 
sense of managing the automation, than fairly autonomous systems such as 
^tOTnaticgrovmd spoilers and automatic rudders. Interestingly, the autoland 
— looS more shnilar in profile to the flight management computer than to 
the automatic rudder. The results of these questions indicate that many 
automated systems provide increased situation awareness without a worldoad 
penalty Tito penalty in obtrusiveness, workload, and the sense of managmg the 
automation that occurs for some components requiring significant pilot- 
component interaction highlights the importance of making pilot interfaces 
these devices as intuitive and simple as possible. 

Aircraft Differences 

There was much evidence that the A-320, MD-11 and B747-400 differ in the level 
and amount of automation they possess. Generally, P d °* 
satisfied with existing automation, and welcome more, particularly for th 
situations in which they are not provided with much automated assistance 
currently There is definitely no indication that there is an "over-automation 
problemwith any of the three aircraft types, though generaUypUotsphdosophy 
StT!o indicate they would be more comfortable with a shared pilot- 
automation performance level than a fully autonomous level of automation 
Pilots' philosophy seemed generally consistent across groups of pilots flying 
dff^ent aircraft types, with the notable exception that A-320 pi ots prefer hard 
£ for^elope^cction and MD-11 and B747-400 pilots prefer soft limits. 

CONCLUSION 

TH* nilnts we have surveyed have presented an interesting portrait of the value 
— tioSThe directions that they wish to see flight deck d«ipi 
take in the next generation of aircraft. They are appreciative of the automation m 
current generation glass cockpits and claim to use it whenever it is a PP™? ria ^ 
^ev w^t dieir automation to be simple and reliable and to produce pre^ctable 
results Not only are these features of automation equated with trustworthy 
automation,* but when the trade-offs are between flexible and adaptable vs 
. I j reliable they still opt for simplicity. Furthermore, m response to 
aSuf*e exLn. to which the P y felt in control of the aircraft vs 
controlling the automation itself, they revealed that simple and reliable is, 
some ways, related to how little attention they need to pay to it. The r^ulte also 
identify the kinds of systems in use today that provide this kind of reliability. 
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T ° 136 a8ainst human-centered a Jordon 
o De against apple pie. The issues instead are to understand from *. » 

££* "T.T £ar ,he * to 8° in intrcSud“madon 

^ ^ nee J to ^ present to maintain situation awareness to assure human 
“ of integrity of flight and to promote ^'and ahHne^ 
ectiveness. The answer provided in these surveys is that the greatest Dromisc 
for further gains is obtained in moving from manual systems tofhaS sv?t^f 
as contrasted with moving from shared control to autonon^L ' 

^ at “ n awareness, in the pilots' opinions, is supported by the varietv of 
P istlc ated navigation, planning and system status displays that are infuse 
today and they report that these are among the most frequently usedZZ ^ 
automation. The majority of pilots surveyed felt thaXbig^ 2Sta 
automation were to further alleviate the mental workload d^mLfds 

the^Zre" dWerencerrf" 01 ' 51 ™' 16 ' 1 deCiSi ° n makin8 situa «ons. Although 
mere were differences of opinion among the pilot populations that had 

expenenee in different aircraft, the similarities were much greater than the 

differences. In the aggregate they indicated the desire for more and higher 

i t 

There is an irony and a challenge in the implications of these two views On iho 
one hand they would like new automation to be s.mple^d^liable bu^hev 

orbusy si toa tions “* m “' ““P* 6 * ^ °‘ ““ i 0 ^ «**»"* Amending 

Finally, a word should be said about the value of this survey from a theoretical 
point of view. In addition to clarifying pilot preferences for future design efforts 
ey gave a boost to efforts by researchers to develop a scientific basis for a 
design philosophy. The survey results should increase confidence that the ha«,v 

tz? ? or the b “ ,dm8 “«*■ tha * 

for the pilot to remain in charge, the impossibility of fore^iny all oroL^ 

Z“ tS ' and 11,6 deSirabi “^ ° f -“■> .ha.ZZfraZrlZ 

° f differCn ' levels °{ au toniation (BiUings, 1991; Regal & Braune 
), while just touched upon in the present survey and in need of further 
refinement, nevertheless was supported by the "clear diffeZttton S 
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fnTrms o? degree of machine intelligence and authonty would be a good 
candidate. 


i out, the spedhc tasxs mat new 

ZESTS' ex^tp “Se ideal level of automation for decis.on making under 
emergency conditions in the landing phase. 

The idea of creating a taxonomy of automation experiences was snpported by 

kSxSs? 

thin! is how they perceive ‘“k * ““So^ISS isTcon«pt 

Itme (Hollnagel, 1991; Wiener, 1989). bu, 
now has a stronger empirical basis. 

vlgariesof statistical analysis, proved to be remarkably robust. 
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Pilot Background Questionnaire 


1. General Information 
Full Name: 

First, Middle, Last 

Address: 

Street and Number, or P.O. Box 


City, State, Zip Code, and Country (if not USA) 


Home Phone: ( ) Work Phone: ( ) 

Area Code Number Area Code Number 

Age 

2. Current Position 

Airline: No. of years 

Seat: No. of years 

Captain, First Officer, Engineer, etc 

Airplane(s): 1. No. of years 

2. No. of years 

3. Past Experience 

Years Flying Commercial (approximate): 

Years Flying Military (approximate): 

Total Hours Flying (approximate): 

Total Hours Flying as Pilot-in-Command (approximate): 

Years of formal education: (e g. high school graduate = 12) 

Please list the aircraft on which you have experience, beginning with that currently flown. Check 
the appropriate boxes under hours in type and simulator hours to indicate the number of hours in 
each. Check the I/CA column if you are/were an instructor (I) or check airman (CA) on an 
aircraft. Check the last column for any aircraft for which you are currently type rated. 


kbBlSHBHI 

Hours in Type 

Simulator Hours 

I/CA 

Currently | 

Type Rated? 1 


<300 

Kiln 

> 1000 

0 

<50 

>50 

Mllll 
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Survey of Pilots of Advanced Automation Aircraft on Philosophy 
Issues Related to Design and Use of Flight Deck Automation 

NASA Langley Research Center 


Section 1 

This se ct i on deals with your experiences with particular automation components. In answering these 
questions, please answer primarily for the aircraft you currently fly. If your aircraft does not have a particular 
automation component, then leave the question blank. Most of the labeling of automation components is based on 
Boeing terminology. If you call the component by another name, please write that label on the questionnaire. If you 
are not sure what component we are trying to identify, please contact one of the researchers at the numbers provided 
on the cover sheet 

Please rate each of the automation components on the scales indicated. The labels at the ends of each scale 
describe the end points of the scale. The midpoint (3) represents a neutral rating. For example, for the first item, if 
ymr attention in flying an aircraft with automatic braking were equally focused on the flight and on the automation, 
you would circle 3. If your attention were slightly more focused on the automation than it was on the flight, you 
would circle 4. When completing statements (b) and (0. which ask for relative information (e.g., higher or lower, 
shallower or deeper), please answer in relation to the case in which you do not have or use that automation 
component 


1. When I use automatic braking: 

(a) My attention is focused on the 

flight 

1 2 

3 

4 

5 

automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though 1 am 

controlling 
the aircraft 

1 2 

3 

4 

5 

managing the 
automation 

(e) I can predict the behavior erf the 

difficulty 

1 2 

3 

4 

5 

ease 

automation with 

(f) My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) In the flight phase or mode where use 

never 

1 2 

3 

4 

5 

always 

of this component is appropriate, I use it 
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2. When I use the automatic engine start: 


(a) My attention is focused on the 

engine 

I 2 

3 

4 

5 

automation 

(b) My understanding of the hig picture is 

shallower 

1 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though I am 

controlling 
the engine 

1 2 

3 

4 

5 

managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 

3 

4 

5 

ease 

(f) My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) In the flight phase or mode where use 

never 

1 2 


A 


always 

of this component is appropriate, I use it 

3 

4 

5 


3. When I use hydraulic amplification of control inputs: 


(a) My attention is focused on the 

flight 

1 2 

3 

4 

5 

automation 

(b) My understanding of the big picture is 

shallower 

I 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 

3 

4 

5 

managing the 
automation 

(c) I can predict the behavior of the 
automation with 

difficulty 

1 2 

3 

4 

5 

ease 

(0 My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) In the flight phase or mode where use 

never 

1 2 


A 

£ 

always 

of this component is appropriate, I use it 

3 

4 

5 


^ When I use a flight director: 


(a) My attention is focused on the 

flight 

1 2 

3 

4 

5 

automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 

3 

4 

5 

managing the 
automation 

(c) I can predict the behavior of the 

automation with 

difficulty 

1 2 

3 

4 

5 

ease 

(0 My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) In the flight phase or mode where use 

never 

1 2 


A 


always 

of this component is appropriate, I use it 

3 

4 

5 
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5. When I use stability augmentation systems: 


(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(f) My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 

When I use a primary flight display: 



(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 

automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(f) My overall workload is 

lower 

1 2 3 4 5 higher 

(g) in the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 

When I use the autopilot to control HDG, SPD, or ALT: 


(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 

automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(f) My overall workload is 

lower 

1 2 3 4 5 higher 

(g) in the flight phase or mode where use never 

of this component is appropriate, I use it 

1 2 3 4 5 always 
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*• Wheu I use the autopilot for pitch, roll, and yaw control: 


(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(0 My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use never 

of this component is appropriate, I use it 

When I use the autopilot to control vertical or horizontal 
navigation paths: 

1 2 3 4 5 always 

(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(f) My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 

When I use the autothrottle: 



(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(f) My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 
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11. When I use automatic landing: 


(a) My attention is focused on the 

flight 

1 2 

3 4 

5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 4 

5 deeper 

(c) The automation is 

unobtrusive 

1 2 

3 4 

5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 

3 4 

5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 

3 4 

5 ease 

(f) My overall workload is 

lower 

1 2 

3 4 

5 higher 

(g) In the flight phase or mode where use 

of this component is appropriate, I use it 

never 

1 2 

3 4 

5 always 

When I use speed envelope limiting: 
(a) My attention is focused on the 

flight 

1 2 

3 4 

3 automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 4 

5 deeper 

(c) The automation is 

unobtrusive 

1 2 

3 4 

5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 

3 4 

5 managing the 
automation 

(e) I can predict the behavior of the 

automation with 

difficulty 

1 2 

3 4 

5 ease 

(0 My overall workload is 

lower 

1 2 

3 4 

5 higher 

(g) in the flight phase or mode where use 

of this component is appropriate, I use it 

never 

1 2 

3 4 

5 always 


13. When I use fly-by-wire engine and/or flight controls: 


(a) My attention is focused on the 

flight 

(b) My understanding of the big picture is 

shallower 

(c) The automation is 

unobtrusive 

(d) I feel as though I am 

controlling 
the aircraft 

(e) I can predict the behavior of the 
automation with 

difficulty 

(0 My overall workload is 

lower 

(g) In the flight phase or mode where use 

of this component is appropriate, I use it 

never 


1 2 3 4 5 automation 

1 2 3 4 5 deeper 

1 2 3 4 5 distracting 

1 2 3 4 5 managing the 
automation 

1 2 3 4 5 ease 

1 2 3 4 5 higher 

1 2 3 4 5 always 
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14. When I use flap limiting (e.g., auto retraction feature): 


(a) My attention is focused on the 

flight 

1 2 3 4 S automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(0 My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 

When I use an auto ground spoiler: 



(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(c) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(0 My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 

When I use automatic compensation for 

asymmetrical thrust (e.g., automatic rudder): 

(a) My attention is focused on the 

flight 

1 2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 2 3 4 5 distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 3 4 5 ease 

(0 My overall workload is 

lower 

1 2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 3 4 5 always 
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17. Whca I use the FMC for automated flight planning (e.g., planning of route, waypoints, 
etc.): 


(a) My attention is focused on the 

flight 

1 2 

3 

4 

5 

automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though I am 

navigating 

l 2 

3 

4 

5 

managing the 
automation 

(e) I can predict the behavior of the 

difficulty 

1 2 

3 

4 

5 

CflSC 

automation with 







(0 My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) in the flight phase or mode where use 

never 

1 2 

3 

4 

5 

always 


of this component is appropriate, I use it 


18. When I use the FMC for automated performance management: 


(a) My attention is focused on the 

flight 

i 

2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

i 

2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

i 

2 3 4 5 distracting 

(d) I feel as though I am 

navigating 

i 

2 3 4 5 managing the 
automation 

(e) 1 can predict the behavior of the 
automation with 

difficulty 

i 

2 3 4 5 ease 

(0 My overall workload is 

lower 

i 

2 3 4 5 higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

i 

2 3 4 5 always 

When I use the FMC for automated flight guidance (e.g.. 

LNAV, VNAV): 

(a) My attention is focused on the 

flight 

1 

2 3 4 5 automation 

(b) My understanding of the big picture is 

shallower 

1 

2 3 4 5 deeper 

(c) The automation is 

unobtrusive 

1 

2 3 4 5 distracting 

(d) I feel as though I am 

navigating 

1 

2 3 4 5 managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 

2 3 4 5 ease 

(f) My overall workload is 

lower 

1 

2 3 4 5 higher 

(g) In the flight phase or mode where use 

never 

1 

2 3 4 5 always 


of this component is appropriate, I use it 
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20. When I use a TCAS traffic display: 

(a) My attention is focused on the 

(b) My understanding of the big picture is 

(c) The automation is 

(d) I feel as though I am 

(e) I can predict the behavior of the 
automation with 

(f) My overall workload is 

(g) In the flight phase or mode where use 

of this component is appropriate, I use it 

21. When I use the navigation display: 

(a) My attention is focused on the 

(b) My understanding of the big picture is 

(c) The automation is 

(d) I feel as though I am 

(c) I can predict the behavior of the 
automation with 

(0 My overall workload is 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 


flight 

1 2 

3 

4 

5 

automation 

shallower 

1 2 

3 

4 

5 

deeper 

unobtrusive 

1 2 

3 

4 

5 

distracting 

navigating 

1 2 

3 

4 

5 

managing the 
automation 

difficulty 

1 2 

3 

4 

5 

ease 

lower 

1 2 

3 

4 

5 

higher 

never 

1 2 

3 

4 

5 

always 


flight 

1 2 

3 

4 

5 

automation 

shallower 

1 2 

3 

4 

5 

deeper 

unobtrusive 

1 2 

3 

4 

5 

distracting 

navigating 

1 2 

3 

4 

5 

managing the 
automation 

difficulty 

1 2 

3 

4 

5 

ease 

lower 

1 2 

3 

4 

5 

higher 

never 

1 2 

3 

4 

5 

always 


22. When I use the inertial reference system: 

(a) My attention is focused on the 

(b) My understanding of the big picture is 

(c) The automation is u 

(d) I feel as though I am . 


(e) I can predict the behavior of the 
automation with 

(0 My overall workload is 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 


1 2 3 4 5 automation 

1 2 3 4 5 deeper 

1 2 3 4 5 distracting 

1 2 3 4 5 managing the 

automation 

1 2 3 4 5 ease 

1 2 3 4 5 higher 

1 2 3 4 5 always 
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23. When I use auto radio tuning: 


(a) My attention is focused on the 

flight 

^ My understanding of the big picture is 

shallower 

(c) The automation is 

unobtrusive 

(d) I feel as though I am communicating 

(e) I can predict the behavior of the 

automation with 

difficulty 

(f) My overall workload is 

lower 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

When I use ACARS: 


(a) My attention is focused on the 

flight 

(b) My understanding of the big picture is 

shallower 

(c) The automation is 

unobtrusive 

(d) 1 feel as though I am 

communicating 

(e) I can predict the behavior of the 
automation with 

difficulty 

(0 My overall workload is 

lower 

(g) In the flight phase or mode where use never 

of this component is appropriate, 1 use it 

When I use sub -systems schematics: 


(a) My attention is focused on the 

flight 

(b) My understanding of the big picture is 

shallower 

(c) The automation is 

unobtrusive 

(d) I feel as though I am 

managing 
the sub-systems 

(e) I can predict the behavior of the 

automation with 

difficulty 

(f) My overall workload is 

lower 

(g) In the flight phase or mode where use never 

of this component is appropriate, I use it 


1 2 3 4 5 automation 

1 2 3 4 5 deeper 

1 2 3 4 5 distracting 

1 2 3 4 5 managing the 

automation 

1 2 3 4 5 ease 

1 2 3 4 5 higher 

1 2 3 4 5 always 

1 2 3 4 5 automation 

1 2 3 4 5 deeper 

1 2 3 4 5 distracting 

1 2 3 4 5 managing the 

automation 

1 2 3 4 5 ease 

1 2 3 4 5 higher 

1 2 3 4 5 always 

1 2 3 4 5 automation 

1 2 3 4 5 deeper 

1 2 3 4 5 distracting 

1 2 3 4 5 managing the 

automation 

1 2 3 4 5 ease 

1 2 3 4 5 higher 

1 2 3 4 5 always 
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26. 


When I use envelope protection with active flight control intervention: 


(a) My attention is focused on the 

flight 

1 2 

3 

4 

5 

automation 

(b) My understanding of the big picture is 

shallower 

1 2 

3 

4 

5 

deeper 

(c) The automation is 

unobtrusive 

1 2 

3 

4 

5 

distracting 

(d) I feel as though I am 

controlling 
the aircraft 

1 2 

3 

4 

5 

managing the 
automation 

(e) I can predict the behavior of the 
automation with 

difficulty 

1 2 

3 

4 

5 

ease 

(f) My overall workload is 

lower 

1 2 

3 

4 

5 

higher 

(g) In the flight phase or mode where use 
of this component is appropriate, I use it 

never 

1 2 

3 

4 

5 

always 


Please complete the following statements concerning alerting systems. 


27. When a master caution and warning alert occurs: 

(a) The alert gets my attention eventually 

(b) I trust that the alert signals a real event never 

(c) I know exactly what the problem is eventually 

(d) My response to the alert requires thought 

(e) Returning to interrupted tasks after 
the alert is 

(0 On my current aircraft, these alerts 
occur 


difficult 

never 


28. When the GPWS warning occurs: 

(a) The alert gets my attention eventually 

(b) I trust that the alert signals a real event never 

(c) I know exactly what the problem is eventually 

(d) My response to the alert requires thought 

(e) Reluming to interrupted tasks after difficult 

the alert is 

(0 On my current aircraft, these alerts never 

occur 


1 2 3 4 5 immediately 
1 2 3 4 5 always 
1 2 3 4 5 immediately 
1 2 3 4 5 is automatic 
1 2 3 4 5 easy 

1 2 3 4 5 very often 

1 2 3 4 5 immediately 
1 2 3 4 5 always 
1 2 3 4 5 immediately 
1 2 3 4 5 is automatic 
1 2 3 4 5 easy 

1 2 3 4 5 very often 
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29. When a configuration warning occurs: 

(a) The alert gets my attention eventually 

(b) I trust that the alert signals a real event never 

(c) I know exactly what the problem is eventually 

(d) My response to the alert requires thought 

(e) Reluming to interrupted tasks after difficult 

the alert is 

(f) On my current aircraft, these alerts 
occur 

30. When a TCAS traffic advisory occurs: 

(a) The alert gets my attention 

(b) I trust that the alert signals a real event 

(c) I know exactly what the problem is 

(d) My response to the alert 

(e) Returning to interrupted tasks after 
the alert is 

(f) On my current aircraft, these alerts 
occur 


never 


eventually 
never 
eventually 
requires thought 
difficult 

never 


31. When a TCAS resolution advisory occurs: 

(a) The alert gets my attention eventually 

(b) I trust that the alert signals a real event never 


(c) I know exactly what the problem is 

(d) My response to the alert 

(e) Returning to interrupted tasks after 

the alert is 

(f) On my current aircraft, these alerts 
occur 


eventually 
requires thought 
difficult 

never 


1 2 3 4 5 immediately 
1 2 3 4 5 always 
1 2 3 4 5 immediately 
1 2 3 4 5 is automatic 
1 2 3 4 5 easy 

1 2 3 4 5 very often 


1 2 3 4 5 immediately 
1 2 3 4 5 always 
1 2 3 4 5 immediately 
1 2 3 4 5 is automatic 
1 2 3 4 5 easy 

1 2 3 4 5 very often 


1 2 3 4 5 immediately 
1 2 3 4 5 always 
1 2 3 4 5 immediately 
1 2 3 4 5 is automatic 
1 2 3 4 5 easy 

1 2 3 4 5 very often 
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Section 2 


This section addresses general automation philosophy issues, including training, pilot and automation roles 
and tasks, decision aids, and automation as it relates to pilot workload, pilot trust, and aircraft safety. 

Please complete each statement by circling the number on the scale that represents the degree to which one 
or the other phrase is consistent with your opinions and beliefs. For example, for the first statement, if you 
believe strongly that initial emphasis should be placed on learning the basic airplane before introducing the 
automation, you would circle "5;" if you believe that the automation should be introduced first, but only 
have a slight preference for this position, you would circle ”2.” 


32. Initial emphasis in training should be placed on learning about: 


the automation before 
introducing the 
"basic airplane" 


the "basic airplane" before 
introducing the 
automation 


1 2 3 4 5 


33, I would like to see the introduction of more automation that: 

assists the pilot in automatically solves 

problem solving problems 

1 2 3 4 5 

34. I would like to see the introduction of more automation that: 


evaluates and advises the 
flight crew on alterna- 
tive plans of action 

1 2 


automatically executes 
alternative plans of 
action 


35. It is likely that situational information about A/C systems given by automation 
in the future: 


will always require 
confirmation 

1 


can always be accepted 
as fact 


36. 


It is likely that procedural information, response recommendations and commands 
concerning A/C systems given by automation in the future: 


will always require 
confirmation 


can always be accepted 
as fact 


37. 


12 3 4 

It is likely that the pilot in the future will: 


not be responsible for 
flying the aircraft 


still be responsible for 
flying the aircraft 


1 
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38. In the future: 


there will still be system 
failures that require devi- 
ations from standard 
procedures 


we can expect that every 
system failure will have 
a prescribed procedure 
to follow 


1 2 3 4 5 

39. The biggest obstacle to total flight safety is: 

system design pilot performance 

1 2 3 4 5 

40. Ii pilot training in the future, principles of navigation and aviation are likely to 

receive: 


less emphasis 

1 2 


just as much emphasis 
4 5 


41. In most cases, an automatic system should: 


prevent the aircraft from 
exceeding its perform- 
ance envelope 


warn the crew of envelope 
exceedence, but not restrict 
pilots' control 


1 2 3 4 5 

42. Autoflight mode annunciation should be organized and displayed by: 


what is controlling (auto- 
throttle, roll, pitch) 

1 2 


what is being controlled (speed, 
lateral path, vert, path) 


43. On my current aircraft, automatic transitions between different autoflight modes 
are usually: 


hard to predict 

1 2 


easy to predict 
4 5 


44. I think synthetic speech for providing information to the pilot should be: 


used only for time-critical 
warnings (e.g., "pull-up," 
windshear") 


used to convey a variety of messages 
(e.g., ATC datalink, FMC, 
ACARS, etc.) 


1 2 3 4 5 

45. I think the overall amount of information available on my aircraft is: 

not enough loo much 

1 2 3 4 5 
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Tlie following questions concern decision aids-automation that provides information, advice, and 
recommendations about possible alternatives. Examples of decision aids would be systems that help 
diagnose a systems fault, select an alternate airport or determine the optimal cost index for the current flight 
conditions. 

44. Decision aids should emphasize: 

situation response 

information information 

1 2 3 4 5 

47. Decision aids should provide the flight crew with: 

one a list of 

alternative alternatives 

1 2 3 4 5 

48. Decision aids should provide the flight crew with: 

recommendations commands 

1 2 3 4 5 

49. Sometimes decision aids weigh evidence and make a probabilistic assessment 
(*•!•» ■ physician's diagnostic aid might determine that your headache, fever, and 
general malaise has an 80% chance of being due to the flu). A decision aid 
should only provide information to the pilot if it is certain that it is 100% 
correct. 

disagree agree 

1 2 3 4 5 

50. If a decision aid tells me the probability of the choice being correct, I will accept 

information that has as low as a % probability of being correct. 
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Automation can be considered good if it enhances safe, economical operatio 
the aircraft. There are many attributes of automation which designers believe 
make it good. Please rank the importance of the 11 attributes listed below for 

good automation. For example, if accountable were the -ost ^Z^ad^tabU 
for good automation, you would put a 1 in the space next to it, and if adapt lab . fe 
were the least important quality for good automation, you would put an 11 in the 
space next to it, and so on. 


Attribute 


Rank 


Accountable means the automation informs the 
pilot of its actions and is able to explain them. 

Adaptable means that displays, control devices, etc., 
are re-programmable within a wide range of pilot 
preferences and needs. 

Comprehensible means that one can figure out 
what the automation is doing and what needs 
to be done to operate it 

Dependable means that the automation does what 
it is supposed to do and never does what it is not 
supposed to do. 

Error-resistant means that the automation keeps 
pilots from committing errors (e.g., disallowing 
in pnu when automation can detect entry is wrong). 

Error-tolerant means that the automation can 
de tect and reduce the effects of errors, given that 
some errors will inevitably occur. 

Flexible means that an appropriate range of modes 
and levels are available to the operator (e.g., from 
manual control to autonomous operation). 

Informative means that the automation imparts 
knowledge to the pilot (e.g., information about the 
airplane, automation, problems, operations, etc.) 

Predictable means that the automation behaves 
as expected (i.e., it is clear what it is going to do ). 

Sim pl e mea n* that it is easy to understand and use 
(i.e., it is straightforward to learn and operate it). 

Subordinate means the automation never assumes 

command, except in pre-defined situations. When 

it does assume command, it can be countermanded. 
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52. Autonation can be cou&ideredtrustworthy if pilots accept it, ase it, and have faith 
in its operation or output. There are many automation attributes which may help 
it be trustworthy. Please rank the importance of the 11 attributes listed below 
for trustworthy automation. For example, if accountable were the most 
important quality for trustworthy automation, you would put a 1 in the space 
next to it, and if adaptable were the least important quality for trustworthy 
automation, you would put an 11 in the space next to it, and so on. 


Attribute 


Rank 


Accountable means the automation informs the 
pilot of its actions and is able to explain them. 

Adaptable means that displays, control devices, etc., 
are re-programmable within a wide range of pilot 
preferences and needs. 

Comprehensible means that one can figure out 
what the automation is doing and what needs 
to be done to operate it 

Dependable means that the automation does what 
it is supposed to do and never does what it is not 
supposed to do. 

Error-resistant means that the automation keeps 
pilots from committing errors (e.g., disallowing 
inputs when automation can delect entry is wrong). 

Error-tolerant means that the automation can 
detect and reduce the effects of errors, given that 
some errors will inevitably occur. 

Flexible means that an appropriate range of modes 
and levels are available to the operator (e.g., from 
manual control to autonomous operation). 

Informative means that the automation imparts 
knowledge to the pilot (e.g., information about the 
airplane, automation, problems, operations, etc.) 

Predictable means that the automation behaves 
as expected (i.e., it is clear what it is going to do ). 

Simple means easy to understand and use (i.e., it 
is straightforward to learn and operate it). 

Subordinate means the automation never assumes 
command, except in pre -defined situations. When 
it does assume command, it can be countermanded. 
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53. In tbe space provided, please describe an experience that exemplifies automation 
that is trustworthy. The experience can be real or imaginary. 



54. Now describe an experience that exemplifies untrustworthy automation. 




55. 


cockpits, relative to coavenfiolial cw^pit^ ha ^ * dvaBced automation 
workload over the flight mission n v P r™ **7* c ^*“* ed the distribution of 
which use mechanical gauge? have i c ™* e " t,0Ml "“Ml*, we mean ones 
DC-g's). By ud»„,c , .d ' .-“I.!" - “.‘.‘ y*** 1 «c. 

computers, «c. (A320's, MD-ll's, 7 “w,) «“< k*«» glass, 


The pilot’s physical workload in 
cockpits, is: 


advanced glass cockpits, relative 


to that in conventional 


Pre-flight 

Taxi 

Take-off 

Climb 

Cruise 

Approach 

Landing 


much 

lower 

1 


2 

2 

2 

2 

2 

2 

2 


about the 
same 

3 

3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 

4 


much 

higher 

5 

5 

5 

5 

5 

5 

5 


The pilot’s mental workload 
cockpits, is: 


in advanced glass cockpits, relative to that in conventional 


Pre-flight 

Taxi 

Take-off 

Climb 

Cruise 

Approach 

Landing 


much 

lower 

1 


2 

2 

2 

2 

2 

2 

2 


about the 
same 

3 

3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 

4 


much 

higher 

5 

5 

5 

5 

5 

5 

5 
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Section 3 

Levels of Automation and Amount of Automation 


• „r . ...niirm n v level of automation, we mean the degree to which 
Questions 56-77 concern levels of „ function, from unassisted human performance, to 

.«*»**>. participates some), to totally automat* 

the automation). 

There are many ways to describe level of/ 

described several of these below. For each <kscnpo reflects your view or opinion. The 

“"r* ? ££ <*“**»■ *“> in,0,VOTe "‘ 

of the automation. 

. tn think about the flight is by normal, non-normal and emergency conditions. Please think 

cumanand ideal levels of automation for each of these categories. 


54. NORMAL FLIGHT (all systems are operating normally, the flight proceeds with no unusual 
events or circumstances) 

IV of mnonMdon ««»^ f°» W 11 ” “* » iCTfl in,is: 


unassisted pilot 
perfonnance 

1 2 

Ideally, the level of automation 1 


totally automated 
performance 


3 4 

would use for normal flight is: 


5 


u nass isted pilot 
perfonnance 


totally automated 
performance 

4 5 


NON-NORMAL FLIGHT (. - ■»««'»*> *- ta k ‘" ,0 “ “* 

diversion to the nearest airport) 

TV madman kvel of auuaaadoa ..ailable for nmowrmnl High. o« Ihe aireraft I fly is: 


unassisted pilot 
performance 

1 2 


totally automated 
performance 


Ideally, the 


level of automation I would use for non-normal flight is: 


unassisted pilot 
perfonnance 

1 2 


totally automated 
perfonnance 
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58. IN-FLIGHT EMERGENCY (tbe aircraft 
airport if possible) 


Bust be landed immediately, at the nearest suitable 


Hie maximum level of Momaioo available to. i~JU,U .m*,**,. M Ue mrcmft I fl, i* 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 

4 5 


Ideally, the level of automation I would use for in-flight emergencies: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 


Please think about «•* «“«*« and ideal level of 


59. PRE-FLIGHT (all activities until aircraft roll-back) 


The maximum level of automation available for pre-flight on the aircraft I fly is: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 


Ideally, the level of automation I would use for pre-flight is: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 


«0. TAXI (roll-back to take-oft roll) 


The maximum level of automation available for taxi on the aircraft I fly is: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 


Ideally, the level of automation I would use for taxi is: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 
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61. 


TAKE-OFF (take-off roll to 500 ft altitude) 

The maximum level of automation available for take-off on the aircraft I fly «s: 


unassisted pilot 
performance 

1 2 3 


totally automated 
performance 


Ideally, the level of automation I would use for take-off is: 


unassisted pilot 
performance 

1 2 


totally automated 
performance 


62. CLIMB (500 ft. attitude to level off at cruise attitude) 

The maximum level of automation available for climb on the aircraft I fly is: 


unassisted pilot 
performance 

1 2 


totally automated 
performance 


Ideally, the level of automation I would use for climb is: 


unassisted pilot 
performance 

1 2 


totally automated 
performance 


63. CRUISE (top of climb to top of descent) 

The maximum level of automation available for cruise on the aircraft 1 fly is: 


unassisted pilot 
performance 

1 


totally automated 
performance 


Ideally, the level of automation I would use for cruise is: 


unassisted pilot 
performance 


totally automated 
performance 


1 
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64. DESCENT (top of descent to 500 ft altitude) 

The maximum level of automation available for descent on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Ideally, the level of automation I would use for descent is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

65. LANDING (500 ft altitude to turn-off at taxi way) 

The maximum level of automation available for landing on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Ideally, the level of automation I would use for landing is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


66. TAXI & PARK (turn-off at taxi way to flight crew deplaning) 

The maximum level of automation available for taxi and park on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Ideally, the level of automation I would use for taxi and park is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 
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Another way to think about the flight is by mission function. Please think about the current and ideal level 
of automation for each mission function. 

67. FLIGHT CONTROL (activities related to controlling the immediate attitude, speed, trajectory, 
and altitude of the aircraft) 

The maximum level of automation available for flight control on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use lot flight control is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

68. NAVIGATION (activities related to planning the (tight path of the aircraft in relation to ATC 
requests, waypoints, destination, etc.) 

The maximum level of navigation on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for navigation is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


69. COMMUNICATION (activities related to transferring information among the flight crew, ATC, 
dispatch, cabin crew, and the FMC) 

The maximum level of automation available for communication on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for communication is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 
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70. SYSTEMS MANAGEMENT (activities related to managing on-board automated systems, 
including fuel, hydraulics, electrical, engines, as well as computer systems such as the FMC. 
autopilot) 

The maximum level of automation available for systems management on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for systems management is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


71. TASK MANAGEMENT (organizing and scheduling tasks to be done during a flight, including 
managing required resources) 

The maximum level of automation available for task management on the aircraft I fly is: 

unassisted pilot totally «mnmar«i 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for task management is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Another way to think about the flight is by human information processing task; taking information in, 
processing it, and responding to it. Please think about the current and ideal level of automation for ea ch 
information processing task. 

72. MONITORING (activities such as scanning, looking, detecting-determining the states and 
status's of the aircraft, systems, etc.) 

The maximum level of automation available for monitoring tasks on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for monitoring tasks is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 
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73. PLANNING (mental activities involving assessing the situation, its consequences, making 
decisions, problem solving, etc) 

The maximum level of automation available for planning tasks on the aircraft I fly is: 

pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for planning tasks is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

74. RESPONDING (determining and performing actions that achieve the plans and goals developed 
in the planning tasks) 

The maximum level of automation for responding task s on the aircraft I fly is. 


M nagsistfri pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for responding tasks is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Another way to think about the flight is by amount of workload involved. Please think about the current 
and ideal level of automation for the different levels of workload described below. 

75. HIGH WORKLOAD SITUATIONS (situations in which you are time-stressed and very 
busy — not sure if you’ll get everything done) 

The maximum level of automation available for high workload situations on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for high workload situations is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


74 



7*. MEDIUM WORKLOAD SITUATIONS (situations in which you are busy but not overwhelmed) 

The maximum level of automation available for medium workload situations 
on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 

Ideally, the level of automation I would use for medium workload situations is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


77. LOW WORKLOAD SITUATIONS (situations in which there is not much to do— obvious 
periods of monotony and inactivity) 

The maximum level of automation available for low workload situations 
on the aircraft I fly is: 

unassisted pilot totally automated 

performance performance 

1 2 3 4 5 


Ideally, the level of automation I would use for low workload situations is: 


unassisted pilot 
performance 


totally automated 
performance 


1 2 3 4 5 
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Items 78-80 list potential advantages and disadvantages of three different levels of flight deck automation: 
autonomous (Q78), shared (Q79), and unassisted (Q80). Please indicate how you feel about the potential 
advantages and disadvantages of each level of automation, as described. 

78. Autonomous operation: automation performs task; pilot may turn on or off, informed of 
malfunction 


I believe that automation at this level holds the promise to: 


Disagree 


keep me involved 1 2 

keep me informed 1 2 

improve my performance 1 2 

improve my situation awareness 1 2 

alleviate fatigue 1 2 

reduce workload 1 2 

provide more precise data 1 2 

increase safety 1 2 

increase airline cost effectiveness 1 2 


3 

3 

3 

3 

3 

3 

3 

3 

3 


I am concerned that automation at this level could lead to: 


Low 

concern 

temperamental devices 1 

display complexity 1 

losing sight of the raw data 1 

difficulty in learning to operate 1 

data entry errors 1 

software engineering errors 1 

need for new skills 1 

unforeseen and unintended 
negative consequences 1 

workload extremes (high and low) 1 

increase in the number of 
alerting signals 1 

loss of situation awareness 1 

need to work around the automatics 
in unusual circumstances 1 

increased head-down time 1 

difficulty in recovering from an 
automation failure 1 

degradation of pilot skills 1 

difficulty in delecting 
system errors 1 

reluctance of crew to take over 
from automatics 1 

complacency 1 


2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


Agree 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 


High 
concern 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 
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I believe that automation at this level holds the promise to: 


keep me involved 
keep me informed 
improve my performance 
improve my situation awareness 
alleviate fatigue 
reduce workload 
provide more precise data 
increase safety 

increase airline cost effectiveness 


Disagree 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 


I am concerned that automation at this level could lead 


temperamental devices 
display complexity 
losing sight of the raw data 
difficulty in learning to operate 
data entry errors 
software engineering errors 
need for new skills 
unforeseen and unintended 
negative consequences 
workload extremes (high and low) 
increase in the number of 
alerting signals 
loss of situation awareness 
need to work around the automatics 
in unusual circumstances 
increased head-down time 
difficulty in recovering from an 
automation failure 
degradation of pilot skills 
difficulty in detecting 
system errors 

reluctance of crew to take over 
from automatics 
complacency 


Low 
concern 
1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

1 2 


3 

3 

3 

3 

3 

3 

3 

3 

3 


to: 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


3 

3 


Agree 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 


High 
concern 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 


77 



80. Unassisted Pilot performance: pilot carries out task; 


; decision-making; utilizes raw data 


I believe that automation at this level holds the promise to: 


Disagree 

keep me involved 1 

keep me informed * 

improve my performance * 

improve my situation awareness 1 

alleviate fatigue 1 

reduce workload ' 

provide more precise data 

increase safety ? - 

increase airline cost effectiveness 1 


2 

2 

2 

2 

2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 

3 

3 

3 


I am concerned that automation at this level could lead to: 


Low 

concern 

temperamental devices 1 

display complexity 1 

losing sight of the raw data 1 

difficulty in learning to operate 1 

data entry errors * 

software engineering errors 1 

need for new skills 1 

unforeseen and unintended 
negative consequences 1 

workload extremes (high and low) 1 

increase in the number of 
alerting signals 1 

loss of situation awareness 1 

need to work around the automatics 
in unusual circum s ta n c e s 1 

increased head-down time 1 

difficulty in recovering from an 
automation failure * 

degradation of pilot skills ^ 

difficulty in detecting 
system errors ' 

rel uct ance of crew to take over 
from automatics 1 

complacency * 


2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 


2 

2 

2 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


3 

3 

3 


Agree 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 


High 
concern 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 

4 5 
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Items 81-85 concern the amount of different types of automation. By amount of automation, we simply 

mean the total number of automated systems or components. 

81. AIRCRAFT CONTROL AUTOMATION (automation that assists or supplants a human pilot in 
guiding the airplane through the maneuvers necessary for aircraft safety.) 

The amount of flight control automation on the aircraft I fly is: 

minimal moderate maximal 

1 2 3 4 5 

The ideal amount of flight control automation in relation to that on the aircraft I fly, is: 

less automation current level more automation 

1 2 3 4 5 

82. SYSTEMS CONTROL AUTOMATION (automation that assists or supplants a human pilot in 

controlling system modes and configurations, display modes and formats, information, etc.) 

The amount of systems control automation on the aircraft I fly is: 

minimal moderate maximal 

1 2 3 4 5 

The ideal amount of systems control automation in relation to that on the aircraft I fly, is 

less automation current level more automation 

1 2 3 4 5 

83. INFORMATION AUTOMATION (automation that informs the pilots about the aircraft and 
systems states, operations, procedures, regulations, and location of the aircraft and relevant 
entities in the environment). 

The amount of information automation on the aircraft I fly is: 

minimal moderate maximal 

1 2 3 4 5 

The ideal amount of information automation in relation to that on the aircraft I fly, is: 

less automation current level more automation 
1 2 3 4 5 
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84. DECISION AUTOMATION (automation that aids the pilot in selecting alternatives, making 
choices). 

The amount of decision a uto m ati on on the aircraft I fly is: 

minimal moderate maximal 

1 2 3 4 5 

The ideal amount of decision automation relation to that on the aircraft I fly, is : 

less automation current level more automation 
1 2 3 4 5 

85. PROTECTIVE AUTOMATION (automation that physically prevents the pilot from taking 
unsafe actions and automatically carries out actions required for safety if the pilot fails to act in 
a timely manner). 

The amount of protective automation on the aircraft I fly is: 

minimal moderate maximal 

1 2 3 4 5 

The ideal amount of protective automation in relation to that on the aircraft I fly, is: 

less automation current level more automation 

1 2 3 4 5 

86. I think is the frer/ automated feature on my 

aircraft because 


87. I think 

aircraft because 


.is the worst automated feature on my 
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Section 4 


NASA is performing research to support development of a Mach 2-3, 6000 mile range, 300 passenger 
commercial aircraft that is economically viable and environmentally sound. Please answer die following 
questions specifically related to such a high speed commercial transport (HSCT). 

It is possible the HSCT will have limited or no forward windows so the cost and weight required to "droop” 
the nose as is done in the Concorde can be saved. Instead, visual information may be provided by sensors, 
by computer-based object and terrain data bases validated by GPS positioning, or by both. The image you 
see may be graphically enhanced to look like an unobstructed, high fidelity, day time view of the outside 
forward-looking scene, possibly augmented with primary flight information. 

Assuming it can be shown that performance with a sensor or computer data base generated visual scene is 
satisfactory and reliable, please indicate your level of agreement with the following statements: 


88. Most pilots would feel comfortable landing a commercial aircraft that supplied a 
forward visual scene from: 


imaging sensors 

computer-based obstacle and 
airport data bases 

a combination of sensor and 
computer-based information 


disagree agree 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 


89. I would feel comfortable landing a commercial aircraft that supplied a forward 
visual scene from: 


imaging sensors 

computer-based obstacle and 
airport data bases 

a combination of sensor and 
computer-based information 


disagree agree 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 


90. I could get used to landing an aircraft that supplied a forward visual scene from: 


imaging sensors 

computer-based obstacle and 
airport data bases 

a combination of sensor and 
computer-based information 


disagree agree 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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91. A forward visual scene generated from sensor and/or computer airport and terrain 
data bases, augmented by symbology like that found on Heads-Up Displays 
(HUD's), will likely be presented on a large field-of-view display. Do you think 
such a display should: 

replace the primary supplement the primary 

flight display flight display 

1 2 3 4 5 

92. I would be more comfortable if sensors used to detect the runway, surface objects 
and aircraft in the terminal area were located: 

on the ground on the aircraft 

1 2 3 4 5 


93. Given that the HSCT will require continuous automated augmentation of the 
pilots' primary control inputs (cables will not be directly linked to control 
surfaces), would you prefer your primary control device to be a: (Check one) 

center stick wheel and column side stick 


94. Which of these control devices have you flown with? (Check all that apply) 
center stick wheel and column side stick 
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Table B1 
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Table B2 


Ordering of Components on Situation Awareness Question (b). Components 
Rated High in Contributing to Situation Awareness are at the Top; 
Components Low in Situation Awareness are at the Bottom. 


Q 

Component 

Score 

21 

navigation display 

4.30 

25 

sub-syst schematics 

4.27 

17 

FMC-flight planning 

4.05 

6 


4.04 

22 

inertial refer system 

3.94 

20 

TCAS traffic display 

3.92 

18 

FMC-auto manage 

3.90 

15 


3.88 

8 

autopilot-pitch roll 

3.85 

9 

autopilot-vrt hz paths 

3.85 

7 

autopilot-HDG SPD ALT 

3.84 

10 

autothrottle 

3.84 

19 

FMCLNAVVNAV 

3.80 

4 

flight director 

3.76 

! 2 

auto engine start 

3.72 

13 

fly-by-wire controls 

3.72 

12 


3.69 

11 

auto landing 

3.63 

23 

auto radio tuning 

3.60 

5 

stability aug systems 

3.57 

24 

ACARS 

3.55 

1 

auto braking 

3.54 

16 

automatic rudder 

3.54 

26 

envelope protection 

3.44 

14 

flap limiting 

3.42 ”1 

3 

hydr amplif of inputs 

3.40 

i — i 1 - 1* / af. _ 


Note . The rating scale used in the questionnaire was the following: 5 = my understanding of the 
big picture is deeper 1= my understanding of the big picture is shallower 
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Table B3 


Ordering of Components on Obtrusiveness Question (c). Components rated 
low in obtrusiveness are at the top; components high in obtrusiveness are at 
the bottom. 


Q 

Component 

Score 

22 

inertial refer system 

4.45 

15 

ground spoiler 

4.44 

3 


4.34 

2 


4.32 

13 

fly-by-wire controls 

4.31 

16 

automatic rudder 

4.28 

5 

stability aug systems 

4.23 

21 


4.23 

1 

auto braking 

4.15 

7 

autopilot-HDG SPD ALT 

4.12 

10 

auto throttle 

4.12 

6 


4.11 

4 

flight director 

4.1 

23 

auto radio timing 

4.09 

25 

sub-syst schematics 

4.09 

18 

FMC-auto manage 

3.92 

14 

flap limiting 

3.84 

12 


3.83 

17 

FMC-flight planning 

3.82 

8 | 

autopilot-pitch roll 

3.8 

9 

autopilot-vrt hz paths 

3.8 

19 

FMC LNAV VNAV 

3.8 

26 


3.72 

11 

auto landing 

3.7 

24 

ACARS 

3.44 

20 

TCAS traffic display 

3.16 


Note . The rating scale has been reversed from the way it appeared in the questionnaire. Here 5 = 
unobtrusive 1* distracting 
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Table B4 


Ordering of Components Controlling/Managing Dimension (d). Components 
Rated High in Controlling are at the Top; Components High in Managing are 
at the Bottom. 


Q 

Component 

Score 

3 

hydr amplif of inputs 

4.10 

15 

auto ground spoiler 

4.00 

13 

fly-by-wire controls 

3.95 

16 

automatic rudder 

3.92 

5 

stability aug systems 

3.87 

6 

primary flight displ 

3.58 

14 

flap limiting 

3.58 

25 

sub-syst schematics 

3.55 

12 

speed envelope limit 

3.47 

24 

ACARS 

3.46 

26 

envelope protection 

3.42 

4 

flight director 

3.38 

20 

TCAS traffic display 

3.38 

22 

inertial refer system 

3.35 

7 

autopilot-HDG SPD AL 

3.33 

10 

auto throttle 

3.33 

21 

navigation display 

3.33 

1 

auto braking 

3.27 

23 

auto radio tuning 

3.02 

8 

autopilot-pitch, roll 

2.75 

9 

autopilot-vrt hz paths 

2.75 

2 

auto engine start 

2.67 

19 

FMC LNAV VNAV 

2.59 

18 

FMC-auto manage 

2.56 

17 

FMC-flight planning 

2.25 

11 

auto landing 

2.02 j 


Note . The rating scale has been reversed from the way it appeared in the questionnaire. Here 5 
= feeling of controlling the flight; 1= feeling of managing the automation 

































































Table 65 


Ordering of Components on Predictability Question (e). Components Rated 
High in Predictability are at the Top; Components Low in Predictability are at 
the Bottom. 


Q 

Component 

Score 

21 

navigation display 

4.3 

15 1 

ground spoiler 

4.2 

22 

inertial refer system 

4.18 

3 

hydr amplif of inputs 

4.17 

2 


4.14 

6 


4.12 

13 

fly-by-wire controls 

4.11 

17 

FMC-flight planning 

4.11 

1 

auto braking 

4.09 

4 1 

flight director 

4.08 

7 

autopilot-HDG SPD ALT 

4.02 

10 

autothrottle 

4.02 

18 

FMC-auto manage 

3.94 

11 

auto landing 

3.91 | 

19 

FMC LN AV VNAV 

3.91 

25 

sub-syst schematics 

3.89 

8 

autopilot-pitch roll 

3.88 

9 

autopilot-vrt hz paths 

3.88 

14 


3.86 

24 

ACARS 

3.86 

16 

automatic rudder 

3.82 

5 

stability aug systems 

3.79 

12 


3.74 

23 

| auto radio tuning 

3.7 1 

26 


3.62 

20 

| TCAS traffic display 

3.2 

. ^ « «« a a« « 1 _ • _ 


Note . The rating scale used in the questionnaire was the following: 5*1 can predict the behavior 
of the automation with ease 1* I can predict the behavior of the automation with difficulty 
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Table B6 


Ordering of Components on Workload Question (f). Components Rated Low 
in Workload are at the Top; Components High in Workload are at the Bottom. 


Q 

Component 

Score 

15 

ground spoiler 

4.21 

10 

autothrottle 

4.14 

22 

inertial refer system 

4.09 

2 

auto engine start 

4.07 

16 

automatic rudder 

4.06 

21 


4.04 

7 

autopilot-HDG SPD ALT 

3.97 1 

3 

hydr amplif of inputs 

3.96 

23 

auto radio tuning 

3.% 

25 

sub-syst schematics 

3.92 

4 

flight director 

3.9 | 

5 

stability aug systems 

3.85 

13 

fly-by-wire controls 

3.81 

8 

autopilot-pitch roll 

3.8 

9 

autopilot-vrt hz paths 

3.8 

1 

auto braking 

3.77 

18 

FMC-auto manage 

3.77 

17 

FMC-flight planning 

| 3.73 

19 

FMC LNAV VNAV 

3.73 

6 


3.7 

12 

■RW55I5TT7? iR 

3.62 

14 

| flap limiting 

3.53 

26 


3.52 

11 

auto landing 

3.51 

24 

ACARS 

3.47 

20 

TCAS traffic display 

3.1 

« • . • .a ♦ V V P 


Note . The rating scale has been reversed from the way it appeared in the questionnaire. Here 5 * 
low workload 1= high workload 


89 





















































Table B7 


Ordering of Components on Frequency of Use Question (g). Components 
Rated High in Use are at the Top; Components Low in Use are at the Bottom. 


Q 

Component 

Score 

22 

inertial refer system 

4.92 

15 

ground spoiler 

4.88 

17 

FMC-flight planning 

4.83 

21 


4.75 

6 


4.73 

13 

fly-by-wire controls 

4.69 

24 

ACARS 

4.69 

5 


4.68 

20 


4.67 

3 

hydr amplif of inputs 

4.63 

25 

sub-syst schematics 

4.59 

2 

auto engine start 

4.58 

4 

flight director 

4.53 

! 18 

FMC-auto manage 

4.52 

23 

auto radio tuning 

4.46 

19 

FMC LNAV VNAV 

4.38 

26 

envelope protection 

4.27 

16 

automatic rudder 

^ 4.26 

12 1 


4.24 

10 

autothrottle 

4.22 

8 

autopilot-pitch roll 

4.16 

9 

autopilot-vrt hz paths 

4.16 

7 

autopilot-HDG SPD ALT 

4.14 

1 

auto braking 

4.13 

14 


3.47 

11 

auto landing j 

2.88 


Note . The rating scale used in the questionnaire was the following: 5 = In the flight phase or 
mode where use of this component is appropriate, I use it always 1= ...I use it never 
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Table 68 


Ordering of Alerts on Attention Getting Question (a). Alerts Rated High in 
Attention Getting are at the Top; Alerts Low in Attention Getting are at the 
Bottom. 


Q 

Alert 

Score 

28 


4.93 

31 

TCAS resolution advisory 

4.93 

30 

TCAS traffic advisory 

4.86 

27 

master caution and warning 

4.83 

29 

configuration warning 

4.82 


Note . The rating scale used in the questionnaire was the following: 5 = The alert gets my 


attention immediately 1= The alert gets my attention eventually 


Table B9 

Ordering of Alerts on Trust Question (b). Alerts Rated High in Trust are at the 
Top; Alerts Low in Trust are at the Bottom. 


Q 1 

Alert 

Score 

29 

configuration warning 

4.53 

31 

TCAS resolution advisory 

4.37 

28 

GPWS warning 

4.26 

27 

master caution and warning 

4.1 

30 

TCAS traffic advisory 

4.09 

Note. The rating scale used in the questionnaire was the following: 5 = i 

trust that the alert 


signals a real event always 1= I trust that the alert signals a real event never 


Table BIO 

Ordering of Alerts on Knowing What the Problem is Question (c). Alerts 
Rated High in Knowing are at the Top; Alerts Low in Knowing are at the 
Bottom. 


Q 

Alert 

Score 

31 

TCAS resolution advisory 

3.93 

29 

configuration warning 

3.76 

30 

TCAS traffic advisory 

3.65 

28 

GPWS warning 

3.59 

27 

master caution and warning 

3.31 

Note. The rating scale used in the questionnaire was the following: 5 = 1 

know exactly what the 


problem is immediately 1=1 know exactly what the problem is eventually 
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Table Bll 


Ordering of Alerts on Automaticity of Response Question (d). Alerts Rated 
High on Automaticity of Response are at the Top; Alerts Low in Automaticity 
are at the Bottom. 


Q 

Alert 

Score 

28 


3.96 

31 

TCAS resolution advisory 

3.91 

29 


3.56 

30 

TCAS traffic advisory 

3.32 

27 

master caution and warning 

2.86 

Note. The rating scale used in the questionnaire was the following: 5 a 

My response to the alert 


is automatic 1 = My response to the alert requires thought 


Table B12 

Ordering of Alerts on Ease of Returning to Interrupted Tasks Question (e). 
Alerts Rated High on Ease of Returning are at the Top; Alerts Low in Ease of 
Returning are at the Bottom. 


Q 

Alert 

Score 

30 

TCAS traffic advisory 

3.64 

29 

configuration warning 

3.63 

27 

master caution and warning 

3.42 

31 

TCAS resolution advisory 

3.40 

28 



3.39 


Note . The rating scale used in the questionnaire was the following: 5 = Returning to interrupted 
tasks after the alert is easy 1 * Returning to interrupted tasks after the alert is difficult 


Table B13 

Ordering of Alerts on Frequency of Alerts Question (f). Alerts Rated High in 
Frequency are at the Top; Alerts Low in Frequency are at the Bottom. 


Q 

Alert 

Score 

30 

TCAS traffic advisory 

2.94 

27 

master caution and warning 

2.7 

31 

TCAS resolution advisory 

2.11 

28 

K*i 

2 

29 

configuration warning 

2 


Note . The rating scale used in the questionnaire was the following: 5 = On my current aircraft. 


these alerts occur very often 1= On my current aircraft, these alerts occur never 
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APPENDIX c 


Discussion and Figures for Miscellaneous Results 
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this is somewhat confusing on current aircraft (see Figur )• 
its primary purpose on current aircraft. 

«-* *« - 

an "information overload" problem on today s flight decks. 

The response; to the agreesUghtiy with 

information by decision aids (Q49 * ^ ^ it f s certa in (100% 

~d « can beseen from Figure C6, the preferred range was 
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si^ts wet l^to assJ how other pilots would feel about 
haWnctflvtdmutT forward window. In question 89 they were ashed how 
they would feel, and in question 90, how they for 
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increased endorsement with each wording change, as the wording became more 
personal and more insistent. The responses changed from the negative end of 
the scale for other pilots (2.69) to fairly positive for self with familiarization (3.63) 
for the combined data sources. These results suggest that oblique questioning 
may be a good way to elicit reservations that pilots might otherwise feel 
uncomfortable about expressing. 

When asked whether they would like a synthetic vision display which provides a 
forward visual scene to replace or supplement the conventional primary flight 
display (Q91), the overwhelming response was that they would like it to 
supplement the primary flight display (mean response = 3.82, where 1 was 
labeled "replace the primary flight display" and 5 was labeled "supplement the 
primary flight display." Over two-thirds of the pilots responded with a "4" or "5" 
(see Figure C8). Pilots also preferred that the sensors used to detect the runway, 
objects, aircraft, etc., for the synthetic vision display be located on the aircraft 
rather than on the ground (Q92, Figure C9). 

Finally, in response to the type of control device they would prefer for an HSCT 
aircraft (Q93), 78 pilots preferred a side stick, 40 preferred a wheel and column, 
and 8 preferred a center stick (see Figure CIO). This result is especially 
interesting given that 98% of the subjects have flown with a wheel and column, 
80% with a center stick, and only 42% with a side stick (Q94). It is not surprising 
that the A-320 pilots chose the side stick since they fly with one, but the B747-400 
pilots preferred it as well (27 preferred the side stick to 15 for the wheel and 
column). The MD-11 pilots were the only ones that preferred a wheel and 
column (23 respondents) to a side-stick (10 respondents); this result may have 
been due to the fact that the MD-11 pilots were older and more resistant to 
change. 
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Figure Cl. Autoflight mode annunciation should be organized and displayed by: 


what is controlling what is being controlled 

(autothrottle, roll, pitch) (speed, lateral path, vert, path) 
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Fieure CZ On my current aircraft, automatictransitions between different autoflight 
modes are usually: 


hard to predict 


easy to predict 
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Figure C3 . I think synthetic speech for providing information to the pilot should be: 


used only for time-critical 
warnings (e.g., “pull-up,” 
“windshear”) 


used to convey a variety 
of messages (e.g., ATC 
datalink, FMC, ACARS, etc.) 
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Figure C4. 1 think the overall amount of information available on my aircraft is: 


not enough 


too much 
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Figure C6. If a decision aid tells me the probability of the choice being correct, I will 

accept information that has as low as a % probability of being correct. 
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Figure C8 
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Ficure C9. I would be more comfortable if sensors used to detect the runway, surface 
objects and aircraft in the terminal area were located: 
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